May 20 15:29:15.417825 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 15:29:15.417836 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 15:29:15.417844 ip-10-0-141-222 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 15:29:15.418135 ip-10-0-141-222 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 15:29:25.477612 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
May 20 15:29:25.477624 ip-10-0-141-222 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e5642d6991ea4c11b50aa4aaa1431702 --
May 20 15:31:41.129363 ip-10-0-141-222 systemd[1]: Starting Kubernetes Kubelet...
May 20 15:31:41.628746 ip-10-0-141-222 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:41.628746 ip-10-0-141-222 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 15:31:41.628746 ip-10-0-141-222 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:41.628746 ip-10-0-141-222 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 15:31:41.628746 ip-10-0-141-222 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:41.630644 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.630560    2574 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 15:31:41.634171 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634151    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:41.634171 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634167    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:41.634171 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634172    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:41.634171 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634176    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634180    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634187    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634193    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634199    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634204    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634208    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634213    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634217    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634221    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634226    2574 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634230    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634234    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634238    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634244    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634248    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634253    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634257    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634261    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:41.634409 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634265    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634269    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634273    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634276    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634280    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634285    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634289    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634294    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634298    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634302    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634306    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634310    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634314    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634319    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634323    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634329    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634334    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634338    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634342    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634346    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:41.635210 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634351    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634356    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634360    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634364    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634370    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634374    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634378    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634382    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634386    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634391    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634394    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634399    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634403    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634406    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634410    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634415    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634419    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634436    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634441    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:41.636062 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634445    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634449    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634453    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634465    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634469    2574 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634473    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634477    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634482    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634488    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634492    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634496    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634500    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634504    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634508    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634512    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634516    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634520    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634524    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634529    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634533    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:41.636888 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634537    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634541    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634545    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634549    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.634553    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635200    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635209    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635214    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635220    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635224    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635229    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635233    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635242    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635246    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635251    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635259    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635264    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635268    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635272    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635276    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:41.637540 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635280    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635285    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635289    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635293    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635297    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635301    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635305    2574 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635309    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635314    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635318    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635322    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635326    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635330    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635334    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635339    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635343    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635347    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635351    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635355    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635358    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:41.638044 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635363    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635367    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635371    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635375    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635379    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635383    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635387    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635391    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635403    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635410    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635415    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635420    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635425    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635430    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635434    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635440    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635443    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635448    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635452    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:41.638644 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635456    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635460    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635464    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635469    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635473    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635476    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635480    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635484    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635489    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635493    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635497    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635501    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635505    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635509    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635513    2574 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635517    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635521    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635525    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635529    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635533    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:41.639316 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635537    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635541    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635549    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635554    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635560    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635565    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635570    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635575    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635580    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635584    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635588    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.635594    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635701    2574 flags.go:64] FLAG: --address="0.0.0.0"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635712    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635720    2574 flags.go:64] FLAG: --anonymous-auth="true"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635726    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635733    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635739    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635745    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635751    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635757    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 15:31:41.639922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635762    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635767    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635773    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635778    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635783    2574 flags.go:64] FLAG: --cgroup-root=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635788    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635793    2574 flags.go:64] FLAG: --client-ca-file=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635798    2574 flags.go:64] FLAG: --cloud-config=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635803    2574 flags.go:64] FLAG: --cloud-provider="external"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635808    2574 flags.go:64] FLAG: --cluster-dns="[]"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635815    2574 flags.go:64] FLAG: --cluster-domain=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635820    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635825    2574 flags.go:64] FLAG: --config-dir=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635834    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635840    2574 flags.go:64] FLAG: --container-log-max-files="5"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635846    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635851    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635856    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635862    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635867    2574 flags.go:64] FLAG: --contention-profiling="false"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635872    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635877    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635882    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635886    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635893    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 15:31:41.640461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635898    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635903    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635908    2574 flags.go:64] FLAG: --enable-load-reader="false"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635913    2574 flags.go:64] FLAG: --enable-server="true"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635918    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635924    2574 flags.go:64] FLAG: --event-burst="100"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635929    2574 flags.go:64] FLAG: --event-qps="50"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635933    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635938    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635943    2574 flags.go:64] FLAG: --eviction-hard=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635949    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635954    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635959    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635964    2574 flags.go:64] FLAG: --eviction-soft=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635969    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635974    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635979    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635984    2574 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635989    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.635994    2574 flags.go:64] FLAG: --fail-swap-on="true"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636000    2574 flags.go:64] FLAG: --feature-gates=""
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636006    2574 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636022    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636028    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636033    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636038    2574 flags.go:64] FLAG: --healthz-port="10248"
May 20 15:31:41.641166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636043    2574 flags.go:64] FLAG: --help="false"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636048    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-141-222.ec2.internal"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636053    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636058    2574 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636062    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636068    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636074    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636078    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636101    2574 flags.go:64] FLAG: --image-service-endpoint=""
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636106    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636110    2574 flags.go:64] FLAG: --kube-api-burst="100"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636115    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636120    2574 flags.go:64] FLAG: --kube-api-qps="50"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636126    2574 flags.go:64] FLAG: --kube-reserved=""
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636131    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636135    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636140    2574 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636145    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636149    2574 flags.go:64] FLAG: --lock-file=""
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636154    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636159    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636164    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636172    2574 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 15:31:41.641825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636177    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636182    2574 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636186    2574 flags.go:64] FLAG: --logging-format="text"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636193    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636198    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636203    2574 flags.go:64] FLAG: --manifest-url=""
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636216    2574 flags.go:64] FLAG: --manifest-url-header=""
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636224    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636229    2574 flags.go:64] FLAG: --max-open-files="1000000"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636234    2574 flags.go:64] FLAG: --max-pods="110"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636240    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636244    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636249    2574 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636254    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636259    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636264    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636269    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636280    2574 flags.go:64] FLAG: --node-status-max-images="50"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636285    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636290    2574 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636295    2574 flags.go:64] FLAG: --pod-cidr=""
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636300    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636309    2574 flags.go:64] FLAG: --pod-manifest-path=""
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636313    2574 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 15:31:41.642407 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636319    2574 flags.go:64] FLAG: --pods-per-core="0"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636323    2574 flags.go:64] FLAG: --port="10250"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636329    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636333    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5d2b00e74bc6283"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636339    2574 flags.go:64] FLAG: --qos-reserved=""
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636344    2574 flags.go:64] FLAG: --read-only-port="10255"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636349    2574 flags.go:64] FLAG: --register-node="true"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636354    2574 flags.go:64] FLAG: --register-schedulable="true"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636359    2574 flags.go:64] FLAG: --register-with-taints=""
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636365    2574 flags.go:64] FLAG: --registry-burst="10"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636370    2574 flags.go:64] FLAG: --registry-qps="5"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636374    2574 flags.go:64] FLAG: --reserved-cpus=""
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636380    2574 flags.go:64] FLAG: --reserved-memory=""
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636386    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636391    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636404    2574 flags.go:64] FLAG: --rotate-certificates="false"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636410    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636414    2574 flags.go:64] FLAG: --runonce="false"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636419    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636424    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636429    2574 flags.go:64] FLAG: --seccomp-default="false"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636434    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636439    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636443    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636449    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636454    2574 flags.go:64] FLAG: --storage-driver-password="root"
May 20 15:31:41.643014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636459    2574 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636463    2574 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636471    2574 flags.go:64] FLAG: --storage-driver-user="root"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636476    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636481    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636486    2574 flags.go:64] FLAG: --system-cgroups=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636491    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636499    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636504    2574 flags.go:64] FLAG: --tls-cert-file=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636508    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636519    2574 flags.go:64] FLAG: --tls-min-version=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636523    2574 flags.go:64] FLAG: --tls-private-key-file=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636528    2574 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636533    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636538    2574 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636543    2574 flags.go:64] FLAG: --v="2"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636549    2574 flags.go:64] FLAG: --version="false"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636555    2574 flags.go:64] FLAG: --vmodule=""
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636563    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.636569    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636785    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636793    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636805    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636810    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:41.643710 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636814    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636819    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636825    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636830    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636834    2574 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636838    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636842    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636846    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636850    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636854    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636861    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636865    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636869    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636873    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636878    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636882    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636886    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636890    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636893    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636897    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:41.644310 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636901    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636905    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636910    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636913    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636917    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636921    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636926    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636931    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636935    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636939    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636943    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636955    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636959    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636964    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636968    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636972    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636976    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636980    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636984    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:41.644806 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636988    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636992    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.636996    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637002    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637006    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637011    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637015    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637019    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637023    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637027    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637031    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637035    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637039    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637043    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637048    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637052    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637056    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637061    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637065    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637070    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:41.645432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637074    2574 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637078    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637098    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637102    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637106    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637118    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637123    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637127    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637133    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637140    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637144    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637148    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637152    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637157    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637162    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637167    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637171    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637176    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637199    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637206    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:41.646119 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637212    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637217    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.637222    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.638044    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.646189    2574 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.646205    2574 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646251    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646255    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646259    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646262    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646265    2574 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646267    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646270    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646273    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646275    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646278    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:41.646625 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646281    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646283    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646286    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646288    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646291    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646294    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646297    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646299    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646302    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646304    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646307    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646309    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646312    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646315    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646317    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646320    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646323    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646325    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646328    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646331    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:41.647057 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646334    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646337    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646340    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646342    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646345    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646347    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646350    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646354    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646358    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646361    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646364    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646367    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646370    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646373    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646375    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646378    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646380    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646383    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646385    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:41.647570 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646388    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646390    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646393    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646395    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646398    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646402    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646405    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646408    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646411    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646413    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646416    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646418    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646421    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646423    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646426    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646429    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646432    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646435    2574 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646438    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646440    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:41.648029 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646443    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646445    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646448    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646450    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646453    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646455    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646458    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646461    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646464    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646467    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646469    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646472    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646475    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646477    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646480    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646482    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:41.648535 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646485    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.646490    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646580    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646585    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646588    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646591    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646594    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646597    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646600    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646602    2574 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646605    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646608    2574 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646610    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646613    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646616    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646618    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:41.648931 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646621    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646623    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646626    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646629    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646631    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646634    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646637    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646639    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646642    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646645    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646648    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646650    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646654    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646658    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646661    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646664    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646666    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646669    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:41.649343 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646673    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646676    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646679    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646682    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646684    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646687    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646689    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646692    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646694    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646697    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646699    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646702    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646704    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646706    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646709    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646712    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646714    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646717    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646720    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646722    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:41.649800 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646725    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646727    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646730    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646733    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646736    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646738    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646741    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646743    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646745    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646748    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646750    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646753    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646755    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646758    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646760    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646762    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646765    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646767    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646770    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646772    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:41.650345 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646775    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646779    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646783    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646787    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646791    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646794    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646796    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646799    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646802    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646804    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646807    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646809    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646812    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:41.646814    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.646819    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:41.650832 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.647633    2574 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 15:31:41.651270 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.651241    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 15:31:41.652299 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.652287    2574 server.go:1019] "Starting client certificate rotation"
May 20 15:31:41.652404 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.652388    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:41.652433 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.652428    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:41.680668 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.680652    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:41.682623 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.682603    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:41.700147 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.700128    2574 log.go:25] "Validated CRI v1 runtime API"
May 20 15:31:41.706441 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.706423    2574 log.go:25] "Validated CRI v1 image API"
May 20 15:31:41.707942 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.707915    2574 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 15:31:41.708917 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.708902    2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:41.713144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.713121    2574 fs.go:135] Filesystem UUIDs: map[1aebf347-7e5e-4c7c-a66f-36bf4899758c:/dev/nvme0n1p3 406ae54f-75a7-4a34-a4d6-9aeccb2f9b8b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 20 15:31:41.713211 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.713143    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 15:31:41.718582 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.718381    2574 manager.go:217] Machine: {Timestamp:2026-05-20 15:31:41.717892481 +0000 UTC m=+0.458643149 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2973294 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26b2c60e66163500acb649728a0b66 SystemUUID:ec26b2c6-0e66-1635-00ac-b649728a0b66 BootID:e5642d69-91ea-4c11-b50a-a4aaa1431702 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:7f:d9:70:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:7f:d9:70:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:df:a0:70:63:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 15:31:41.719393 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.719381    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 15:31:41.719470 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.719458    2574 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 15:31:41.722508 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.722480    2574 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 15:31:41.722654 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.722527    2574 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 15:31:41.722695 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.722664    2574 topology_manager.go:138] "Creating topology manager with none policy"
May 20 15:31:41.722695 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.722673    2574 container_manager_linux.go:306] "Creating device plugin manager"
May 20 15:31:41.722695 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.722679    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:41.723544 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.723534    2574 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:41.726530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.726519    2574 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:41.726640 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.726631    2574 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 15:31:41.729302 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.729293    2574 kubelet.go:491] "Attempting to sync node with API server"
May 20 15:31:41.729337 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.729305    2574 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 15:31:41.729337 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.729317    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 15:31:41.729337 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.729329    2574 kubelet.go:397] "Adding apiserver pod source"
May 20 15:31:41.729337 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.729337    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 15:31:41.730417 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.730397    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2pj56"
May 20 15:31:41.730671 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.730660    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:41.730702 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.730678    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:41.733989 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.733972    2574 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 15:31:41.736569 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.736556    2574 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 15:31:41.737720 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737691    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 15:31:41.737720 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737722    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737737    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737750    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737765    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737776    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737787    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737798    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737810    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737820    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737687    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2pj56"
May 20 15:31:41.737869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.737848    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 15:31:41.738212 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.738200    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 15:31:41.739172 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.739159    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 15:31:41.739240 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.739179    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 15:31:41.742324 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.742307    2574 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 15:31:41.742417 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.742352    2574 server.go:1295] "Started kubelet"
May 20 15:31:41.742484 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.742438    2574 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 15:31:41.742528 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.742504    2574 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 15:31:41.742565 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.742458    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 15:31:41.743138 ip-10-0-141-222 systemd[1]: Started Kubernetes Kubelet.
May 20 15:31:41.744642 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.744615    2574 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 15:31:41.745024 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.745008    2574 server.go:317] "Adding debug handlers to kubelet server"
May 20 15:31:41.745129 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.745116    2574 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:41.748366 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.748346    2574 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:41.753961 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.753937    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-222.ec2.internal" not found
May 20 15:31:41.754522 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.754499    2574 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 15:31:41.755390 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.755369    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 15:31:41.756016 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756003    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 15:31:41.756365 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756336    2574 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 15:31:41.756365 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756364    2574 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 15:31:41.756620 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756603    2574 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 15:31:41.756707 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756692    2574 reconstruct.go:97] "Volume reconstruction finished"
May 20 15:31:41.756707 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.756702    2574 reconciler.go:26] "Reconciler: start to sync state"
May 20 15:31:41.757172 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757155    2574 factory.go:55] Registering systemd factory
May 20 15:31:41.757232 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757184    2574 factory.go:223] Registration of the systemd container factory successfully
May 20 15:31:41.757439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757425    2574 factory.go:153] Registering CRI-O factory
May 20 15:31:41.757497 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757444    2574 factory.go:223] Registration of the crio container factory successfully
May 20 15:31:41.757532 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757516    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 15:31:41.757579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757536    2574 factory.go:103] Registering Raw factory
May 20 15:31:41.757579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.757554    2574 manager.go:1196] Started watching for new ooms in manager
May 20 15:31:41.758009 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.757967    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found"
May 20 15:31:41.759118 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.759103    2574 manager.go:319] Starting recovery of all containers
May 20 15:31:41.759955 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.759936    2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:41.762298 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.762275    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-222.ec2.internal\" not found" node="ip-10-0-141-222.ec2.internal"
May 20 15:31:41.769121 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.769104    2574 manager.go:324] Recovery completed
May 20 15:31:41.771239 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.771225    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-222.ec2.internal" not found
May 20 15:31:41.773024 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.773013    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:41.774777 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.774762    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:41.774849 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.774789    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:41.774849 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.774799    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:41.775239 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.775225    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 15:31:41.775239 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.775237    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 15:31:41.775328 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.775254    2574 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:41.778135 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.778120    2574 policy_none.go:49] "None policy: Start"
May 20 15:31:41.778223 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.778139    2574 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 15:31:41.778223 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.778153    2574 state_mem.go:35] "Initializing new in-memory state store"
May 20 15:31:41.822641 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.822627    2574 manager.go:341] "Starting Device Plugin manager"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.822666    2574 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.822676    2574 server.go:85] "Starting device plugin registration server"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.823386    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.823402    2574 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.823499    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.823599    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.823606    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.824029    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 15:31:41.824730 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.824067    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-222.ec2.internal\" not found"
May 20 15:31:41.832147 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.832132    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-222.ec2.internal" not found
May 20 15:31:41.855808 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.855789    2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 15:31:41.856946 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.856926    2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 15:31:41.857013 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.856960    2574 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 15:31:41.857013 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.856975    2574 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 15:31:41.857013 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.856983    2574 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 15:31:41.857138 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.857022    2574 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 15:31:41.860249 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.860230    2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:41.924484 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.924436    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:41.925453 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.925438    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:41.925529 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.925466    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:41.925529 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.925477    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:41.925529 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.925504    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-222.ec2.internal"
May 20 15:31:41.934952 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.934940    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-222.ec2.internal"
May 20 15:31:41.934995 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:41.934958    2574 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-222.ec2.internal\": node \"ip-10-0-141-222.ec2.internal\" not found"
May 20 15:31:41.958112 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.958073    2574 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"]
May 20 15:31:41.960625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.960612    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:41.960709 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.960613    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:41.986628 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.986613    2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:41.991620 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.991600    2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:41.997455 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:41.997439    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:42.003061 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.003046    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:42.057629 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.057608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5244faf83b4a2df37ec043783dad277-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"a5244faf83b4a2df37ec043783dad277\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.057721 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.057632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.057721 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.057649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158438 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158410    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158438 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158440    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158617 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158458    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5244faf83b4a2df37ec043783dad277-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"a5244faf83b4a2df37ec043783dad277\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158617 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158512    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158617 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158514    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3aec373d631fee7d033910fa781e961-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"a3aec373d631fee7d033910fa781e961\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.158617 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.158553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5244faf83b4a2df37ec043783dad277-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"a5244faf83b4a2df37ec043783dad277\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.301357 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.301282    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.306932 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.306911    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal"
May 20 15:31:42.652209 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.652187    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 15:31:42.652820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.652309    2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:42.652820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.652359    2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:42.652820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.652360    2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:42.729824 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.729795    2574 apiserver.go:52] "Watching apiserver"
May 20 15:31:42.735718 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.735696    2574 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 15:31:42.737300 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.737281    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2gv4s","openshift-ovn-kubernetes/ovnkube-node-tjrxd","kube-system/konnectivity-agent-gdjgt","openshift-cluster-node-tuning-operator/tuned-wnpr5","openshift-multus/multus-p2rvk","openshift-network-diagnostics/network-check-target-dz424","kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g","openshift-image-registry/node-ca-cjl8v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal","openshift-multus/multus-additional-cni-plugins-ct8vp","openshift-multus/network-metrics-daemon-qmkpv"]
May 20 15:31:42.739055 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.739037    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.740245 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.740213    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 15:26:41 +0000 UTC" deadline="2028-01-19 05:52:50.916433566 +0000 UTC"
May 20 15:31:42.740245 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.740241    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14606h21m8.176195649s"
May 20 15:31:42.740394 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.740376    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.741389 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.741371    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:42.741389 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.741382    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 15:31:42.741518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.741378    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j66cd\""
May 20 15:31:42.741518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.741377    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 15:31:42.741856 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.741840    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.742570 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.742547    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 15:31:42.742674 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.742657    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 15:31:42.742725 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.742675    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 15:31:42.742725 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.742689    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-764gr\""
May 20 15:31:42.743166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743147    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 15:31:42.743267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743169    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.743267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743176    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 15:31:42.743267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743237    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 15:31:42.743749 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743735    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 15:31:42.743848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743833    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 15:31:42.743908 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.743873    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55f6w\""
May 20 15:31:42.744505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.744489    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:42.744588 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.744551    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:42.745181 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745165    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 15:31:42.745260 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745246    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 15:31:42.745309 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745246    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 15:31:42.745356 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745331    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xtdrr\""
May 20 15:31:42.745408 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745366    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 15:31:42.745765 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.745750    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.747270 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.747254    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.747582 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.747560    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 15:31:42.747698 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.747683    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 15:31:42.747797 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.747779    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rjzrw\""
May 20 15:31:42.747893 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.747873    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 15:31:42.748567 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.748550    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.749965 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.749482    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:42.749965 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.749613    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 15:31:42.750156 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.750043    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttprt\""
May 20 15:31:42.750398 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.750337    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.751166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.750699    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 15:31:42.751166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.750727    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 15:31:42.751166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.750933    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pt4bm\""
May 20 15:31:42.751166 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.751049    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 15:31:42.752995 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.752976    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.753129 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.753098    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:42.753581 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.753493    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 15:31:42.753786 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.753739    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 15:31:42.753786 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.753756    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zx8cc\""
May 20 15:31:42.755500 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.755481    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 15:31:42.757859 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.757845    2574 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 15:31:42.762926 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-bin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-multus\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-etc-kubernetes\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762979    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-script-lib\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.762996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-kubelet\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfvc\" (UniqueName: \"kubernetes.io/projected/f1736103-41b1-47b5-beac-d9442dd46cc0-kube-api-access-tvfvc\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-sys-fs\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763157    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-tmp\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-system-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-kubelet\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-modprobe-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fd9469-a348-412a-abf5-c8ef100edb5e-host-slash\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763319    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895zq\" (UniqueName: \"kubernetes.io/projected/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-kube-api-access-895zq\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-systemd\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2edd435-0b32-47a9-91ef-48736be5945b-agent-certs\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1736103-41b1-47b5-beac-d9442dd46cc0-host\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-netd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.763439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-slash\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-etc-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-env-overrides\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-os-release\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-conf\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-host\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4fd9469-a348-412a-abf5-c8ef100edb5e-iptables-alerter-script\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-node-log\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cnibin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-etc-tuned\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-socket-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-registration-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqsv7\" (UniqueName: \"kubernetes.io/projected/1a472476-846f-4b44-aa82-002fa5ca7715-kube-api-access-qqsv7\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-system-cni-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-cnibin\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.763790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-os-release\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-kubernetes\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-run\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-systemd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-daemon-config\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-multus-certs\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysconfig\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.763986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2rq\" (UniqueName: \"kubernetes.io/projected/e4fd9469-a348-412a-abf5-c8ef100edb5e-kube-api-access-cq2rq\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-systemd-units\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-netns\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn78n\" (UniqueName: \"kubernetes.io/projected/43619788-ee3e-4703-a24c-29887c78e05f-kube-api-access-fn78n\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764100    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-netns\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764123    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-conf-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764145    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-sys\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.764563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-log-socket\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1736103-41b1-47b5-beac-d9442dd46cc0-serviceca\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cni-binary-copy\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-socket-dir-parent\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-k8s-cni-cncf-io\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-hostroot\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth49\" (UniqueName: \"kubernetes.io/projected/9e4f292c-23eb-45d1-be14-7e368f49ff38-kube-api-access-lth49\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-var-lib-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c34c181-6934-4c86-bf7d-71221abd751c-ovn-node-metrics-cert\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-var-lib-kubelet\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jp2g\" (UniqueName: \"kubernetes.io/projected/857b75bc-eba1-4597-a9bb-9855b531519c-kube-api-access-4jp2g\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-bin\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-config\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2pc\" (UniqueName: \"kubernetes.io/projected/4c34c181-6934-4c86-bf7d-71221abd751c-kube-api-access-br2pc\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764629    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2edd435-0b32-47a9-91ef-48736be5945b-konnectivity-ca\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-lib-modules\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-ovn\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.765539 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.764754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-device-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.767793 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.767776    2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:42.780633 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:42.780605    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3aec373d631fee7d033910fa781e961.slice/crio-8dbd5a5fc32b8f35ce02d8c1489e1934416908cab32d89eaaea7b20e45bc6110 WatchSource:0}: Error finding container 8dbd5a5fc32b8f35ce02d8c1489e1934416908cab32d89eaaea7b20e45bc6110: Status 404 returned error can't find the container with id 8dbd5a5fc32b8f35ce02d8c1489e1934416908cab32d89eaaea7b20e45bc6110
May 20 15:31:42.780886 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:42.780868    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5244faf83b4a2df37ec043783dad277.slice/crio-ed767555ef5cc4c10ac3bc49dae7f578cf84a377bf4c3bbba85ae5c06ef0d16b WatchSource:0}: Error finding container ed767555ef5cc4c10ac3bc49dae7f578cf84a377bf4c3bbba85ae5c06ef0d16b: Status 404 returned error can't find the container with id ed767555ef5cc4c10ac3bc49dae7f578cf84a377bf4c3bbba85ae5c06ef0d16b
May 20 15:31:42.785244 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.785227    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:31:42.799254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.799237    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9n72r"
May 20 15:31:42.805593 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.805566    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9n72r"
May 20 15:31:42.859603 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.859566    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"a5244faf83b4a2df37ec043783dad277","Type":"ContainerStarted","Data":"ed767555ef5cc4c10ac3bc49dae7f578cf84a377bf4c3bbba85ae5c06ef0d16b"}
May 20 15:31:42.863162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.863139    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"a3aec373d631fee7d033910fa781e961","Type":"ContainerStarted","Data":"8dbd5a5fc32b8f35ce02d8c1489e1934416908cab32d89eaaea7b20e45bc6110"}
May 20 15:31:42.865022 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865002    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-config\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865074 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865032    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-br2pc\" (UniqueName: \"kubernetes.io/projected/4c34c181-6934-4c86-bf7d-71221abd751c-kube-api-access-br2pc\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865074 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865048    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2edd435-0b32-47a9-91ef-48736be5945b-konnectivity-ca\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.865074 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865063    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865078    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-lib-modules\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865111    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865126    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-ovn\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865147    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-device-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865175    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865200    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-bin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865209    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-device-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.865227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865223    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-multus\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865246    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-etc-kubernetes\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.865251    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865269    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-script-lib\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865294    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-kubelet\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.865341    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:43.365302524 +0000 UTC m=+2.106053203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-kubelet\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865360    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfvc\" (UniqueName: \"kubernetes.io/projected/f1736103-41b1-47b5-beac-d9442dd46cc0-kube-api-access-tvfvc\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-etc-kubernetes\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865389    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-sys-fs\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865423    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865425    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-bin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865448    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-tmp\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865473    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-system-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865499    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-kubelet\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865523    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-modprobe-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865549    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fd9469-a348-412a-abf5-c8ef100edb5e-host-slash\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.865608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865585    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865617    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-895zq\" (UniqueName: \"kubernetes.io/projected/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-kube-api-access-895zq\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865626    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2edd435-0b32-47a9-91ef-48736be5945b-konnectivity-ca\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865643    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-systemd\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865672    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2edd435-0b32-47a9-91ef-48736be5945b-agent-certs\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865696    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1736103-41b1-47b5-beac-d9442dd46cc0-host\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865705    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-kubelet\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865712    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865722    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-netd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865758    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-sys-fs\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865760    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865801    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fd9469-a348-412a-abf5-c8ef100edb5e-host-slash\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865805    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-slash\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865813    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865816    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-config\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865836    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-slash\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865842    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-etc-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.866375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865853    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-ovnkube-script-lib\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865838    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865899    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-netd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865908    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-system-cni-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865940    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-env-overrides\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-etc-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865968    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-os-release\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865976    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-modprobe-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.865992    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866019    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-conf\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866029    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-lib-modules\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-os-release\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866041    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-host\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866041    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-systemd\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866065    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1736103-41b1-47b5-beac-d9442dd46cc0-host\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866078    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866098    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-host\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866118    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-ovn\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866120    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4fd9469-a348-412a-abf5-c8ef100edb5e-iptables-alerter-script\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866148    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-node-log\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866145    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-var-lib-cni-multus\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866160    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-conf\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866187    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-node-log\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866225    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cnibin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866252    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-etc-tuned\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866275    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-socket-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866302    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-registration-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cnibin\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866336    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqsv7\" (UniqueName: \"kubernetes.io/projected/1a472476-846f-4b44-aa82-002fa5ca7715-kube-api-access-qqsv7\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866359    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c34c181-6934-4c86-bf7d-71221abd751c-env-overrides\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866384    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-registration-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866387    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866430    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-system-cni-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-system-cni-dir\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.867951 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-socket-dir\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866459    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-cnibin\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43619788-ee3e-4703-a24c-29887c78e05f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866484    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-os-release\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43619788-ee3e-4703-a24c-29887c78e05f-cnibin\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866514    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-kubernetes\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866544    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-run\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866565    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-kubernetes\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866569    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866593    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866612    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-run\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866620    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-systemd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866635    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866645    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-daemon-config\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866568    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-os-release\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866671    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a472476-846f-4b44-aa82-002fa5ca7715-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866670    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-multus-certs\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866693    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-systemd\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.868703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866727    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-multus-certs\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866753    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysconfig\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866778    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2rq\" (UniqueName: \"kubernetes.io/projected/e4fd9469-a348-412a-abf5-c8ef100edb5e-kube-api-access-cq2rq\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866793    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-systemd-units\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866808    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-netns\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866836    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fn78n\" (UniqueName: \"kubernetes.io/projected/43619788-ee3e-4703-a24c-29887c78e05f-kube-api-access-fn78n\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysconfig\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866860    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-netns\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866900    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-conf-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866923    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-sys\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866948    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866973    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-log-socket\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866998    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1736103-41b1-47b5-beac-d9442dd46cc0-serviceca\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867007    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-netns\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867020    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cni-binary-copy\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867045    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-socket-dir-parent\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867071    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-k8s-cni-cncf-io\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867110    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-hostroot\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e4fd9469-a348-412a-abf5-c8ef100edb5e-iptables-alerter-script\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867135    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-daemon-config\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867177    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-sys\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867120    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-conf-dir\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867177    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-run-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867169    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lth49\" (UniqueName: \"kubernetes.io/projected/9e4f292c-23eb-45d1-be14-7e368f49ff38-kube-api-access-lth49\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867238    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-var-lib-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867246    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-run-netns\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.866948    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-systemd-units\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867264    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-log-socket\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867302    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-var-lib-openvswitch\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867306    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-host-run-k8s-cni-cncf-io\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867263    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c34c181-6934-4c86-bf7d-71221abd751c-ovn-node-metrics-cert\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867348    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867376    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-var-lib-kubelet\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-etc-sysctl-d\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.869843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857b75bc-eba1-4597-a9bb-9855b531519c-var-lib-kubelet\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867400    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jp2g\" (UniqueName: \"kubernetes.io/projected/857b75bc-eba1-4597-a9bb-9855b531519c-kube-api-access-4jp2g\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867653    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-bin\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867683    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867708    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1736103-41b1-47b5-beac-d9442dd46cc0-serviceca\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867749    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-multus-socket-dir-parent\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c34c181-6934-4c86-bf7d-71221abd751c-host-cni-bin\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.867455    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-hostroot\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.868098    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-cni-binary-copy\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.868960    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-etc-tuned\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.868999    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857b75bc-eba1-4597-a9bb-9855b531519c-tmp\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.869126    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2edd435-0b32-47a9-91ef-48736be5945b-agent-certs\") pod \"konnectivity-agent-gdjgt\" (UID: \"c2edd435-0b32-47a9-91ef-48736be5945b\") " pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:42.870303 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.869745    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c34c181-6934-4c86-bf7d-71221abd751c-ovn-node-metrics-cert\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:42.873975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.873954    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfvc\" (UniqueName: \"kubernetes.io/projected/f1736103-41b1-47b5-beac-d9442dd46cc0-kube-api-access-tvfvc\") pod \"node-ca-cjl8v\" (UID: \"f1736103-41b1-47b5-beac-d9442dd46cc0\") " pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:42.874039 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.874005    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895zq\" (UniqueName: \"kubernetes.io/projected/e4d43c9f-a1da-4bb9-afe8-ad383db25c79-kube-api-access-895zq\") pod \"multus-p2rvk\" (UID: \"e4d43c9f-a1da-4bb9-afe8-ad383db25c79\") " pod="openshift-multus/multus-p2rvk"
May 20 15:31:42.877043 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.877024    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:42.877146 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.877046    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:42.877146 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.877058    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:42.877223 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:42.877169    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:43.377151393 +0000 UTC m=+2.117902069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:42.878995 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.878969    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqsv7\" (UniqueName: \"kubernetes.io/projected/1a472476-846f-4b44-aa82-002fa5ca7715-kube-api-access-qqsv7\") pod \"aws-ebs-csi-driver-node-s2t8g\" (UID: \"1a472476-846f-4b44-aa82-002fa5ca7715\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:42.879198 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.879179    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jp2g\" (UniqueName: \"kubernetes.io/projected/857b75bc-eba1-4597-a9bb-9855b531519c-kube-api-access-4jp2g\") pod \"tuned-wnpr5\" (UID: \"857b75bc-eba1-4597-a9bb-9855b531519c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:42.879307 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.879289    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2rq\" (UniqueName: \"kubernetes.io/projected/e4fd9469-a348-412a-abf5-c8ef100edb5e-kube-api-access-cq2rq\") pod \"iptables-alerter-2gv4s\" (UID: \"e4fd9469-a348-412a-abf5-c8ef100edb5e\") " pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:42.879528 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.879511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth49\" (UniqueName: \"kubernetes.io/projected/9e4f292c-23eb-45d1-be14-7e368f49ff38-kube-api-access-lth49\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:42.879571 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.879558    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn78n\" (UniqueName: \"kubernetes.io/projected/43619788-ee3e-4703-a24c-29887c78e05f-kube-api-access-fn78n\") pod \"multus-additional-cni-plugins-ct8vp\" (UID: \"43619788-ee3e-4703-a24c-29887c78e05f\") " pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:42.879990 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:42.879976    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2pc\" (UniqueName: \"kubernetes.io/projected/4c34c181-6934-4c86-bf7d-71221abd751c-kube-api-access-br2pc\") pod \"ovnkube-node-tjrxd\" (UID: \"4c34c181-6934-4c86-bf7d-71221abd751c\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:43.058080 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.057921    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2gv4s"
May 20 15:31:43.064669 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.064646    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fd9469_a348_412a_abf5_c8ef100edb5e.slice/crio-6b41bf02fd4ea62353dfd3f19e61136601b38c78c167fcec44e84cb90cc56138 WatchSource:0}: Error finding container 6b41bf02fd4ea62353dfd3f19e61136601b38c78c167fcec44e84cb90cc56138: Status 404 returned error can't find the container with id 6b41bf02fd4ea62353dfd3f19e61136601b38c78c167fcec44e84cb90cc56138
May 20 15:31:43.073295 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.073275    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:31:43.078584 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.078560    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c34c181_6934_4c86_bf7d_71221abd751c.slice/crio-35d72dd0ad7d0fea6d1ce0f9f19b0307f84c1cbff5b97f4060a9e6175754ca16 WatchSource:0}: Error finding container 35d72dd0ad7d0fea6d1ce0f9f19b0307f84c1cbff5b97f4060a9e6175754ca16: Status 404 returned error can't find the container with id 35d72dd0ad7d0fea6d1ce0f9f19b0307f84c1cbff5b97f4060a9e6175754ca16
May 20 15:31:43.089198 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.089181    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:31:43.093712 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.093695    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2rvk"
May 20 15:31:43.094963 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.094940    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2edd435_0b32_47a9_91ef_48736be5945b.slice/crio-afcd12cd6410ed2984f6121e5860f869ed723fcae6347f553cd8423ee21c2452 WatchSource:0}: Error finding container afcd12cd6410ed2984f6121e5860f869ed723fcae6347f553cd8423ee21c2452: Status 404 returned error can't find the container with id afcd12cd6410ed2984f6121e5860f869ed723fcae6347f553cd8423ee21c2452
May 20 15:31:43.098211 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.098109    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cjl8v"
May 20 15:31:43.099485 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.099467    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d43c9f_a1da_4bb9_afe8_ad383db25c79.slice/crio-62a9f54ce092345071c279a8b4183ba49e12a5df9780d62d61f8c0a145f8b039 WatchSource:0}: Error finding container 62a9f54ce092345071c279a8b4183ba49e12a5df9780d62d61f8c0a145f8b039: Status 404 returned error can't find the container with id 62a9f54ce092345071c279a8b4183ba49e12a5df9780d62d61f8c0a145f8b039
May 20 15:31:43.103855 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.103834    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wnpr5"
May 20 15:31:43.104131 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.104110    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1736103_41b1_47b5_beac_d9442dd46cc0.slice/crio-cedb1f0662e7e762565587ec197f17b113e3c438640d89429f35ba0c15f7fb57 WatchSource:0}: Error finding container cedb1f0662e7e762565587ec197f17b113e3c438640d89429f35ba0c15f7fb57: Status 404 returned error can't find the container with id cedb1f0662e7e762565587ec197f17b113e3c438640d89429f35ba0c15f7fb57
May 20 15:31:43.109359 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.109340    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857b75bc_eba1_4597_a9bb_9855b531519c.slice/crio-403e9f7523c6c1381247516a7c801d2f37af8dd154f099258bf7bc66cbb877bf WatchSource:0}: Error finding container 403e9f7523c6c1381247516a7c801d2f37af8dd154f099258bf7bc66cbb877bf: Status 404 returned error can't find the container with id 403e9f7523c6c1381247516a7c801d2f37af8dd154f099258bf7bc66cbb877bf
May 20 15:31:43.109437 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.109410    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g"
May 20 15:31:43.114527 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.114506    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ct8vp"
May 20 15:31:43.114745 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.114716    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a472476_846f_4b44_aa82_002fa5ca7715.slice/crio-07e1bd9a162a8d15b116779e7f4ea8a6eeca4ee21c7b0ca1e89460b4acf5e811 WatchSource:0}: Error finding container 07e1bd9a162a8d15b116779e7f4ea8a6eeca4ee21c7b0ca1e89460b4acf5e811: Status 404 returned error can't find the container with id 07e1bd9a162a8d15b116779e7f4ea8a6eeca4ee21c7b0ca1e89460b4acf5e811
May 20 15:31:43.121046 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:31:43.121025    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43619788_ee3e_4703_a24c_29887c78e05f.slice/crio-262e018990f02660da94bbe3c7626ba0585a6bc22d16a2e01785282da5efb123 WatchSource:0}: Error finding container 262e018990f02660da94bbe3c7626ba0585a6bc22d16a2e01785282da5efb123: Status 404 returned error can't find the container with id 262e018990f02660da94bbe3c7626ba0585a6bc22d16a2e01785282da5efb123
May 20 15:31:43.371320 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.371290    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:43.371492 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.371406    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:43.371492 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.371459    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:44.3714449 +0000 UTC m=+3.112195568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:43.471844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.471813    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:43.472013 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.471960    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:43.472013 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.471979    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:43.472013 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.471990    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:43.472189 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.472044    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:44.47202556 +0000 UTC m=+3.212776219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:43.491971 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.491944    2574 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:43.807281 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.807155    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:42 +0000 UTC" deadline="2028-01-11 20:35:27.960202088 +0000 UTC"
May 20 15:31:43.807281 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.807187    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14429h3m44.153018101s"
May 20 15:31:43.858185 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.857654    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:43.858185 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:43.857777    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:43.876844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.876809    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerStarted","Data":"262e018990f02660da94bbe3c7626ba0585a6bc22d16a2e01785282da5efb123"}
May 20 15:31:43.880879 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.880764    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2rvk" event={"ID":"e4d43c9f-a1da-4bb9-afe8-ad383db25c79","Type":"ContainerStarted","Data":"62a9f54ce092345071c279a8b4183ba49e12a5df9780d62d61f8c0a145f8b039"}
May 20 15:31:43.889968 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.889940    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdjgt" event={"ID":"c2edd435-0b32-47a9-91ef-48736be5945b","Type":"ContainerStarted","Data":"afcd12cd6410ed2984f6121e5860f869ed723fcae6347f553cd8423ee21c2452"}
May 20 15:31:43.893760 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.893734    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"35d72dd0ad7d0fea6d1ce0f9f19b0307f84c1cbff5b97f4060a9e6175754ca16"}
May 20 15:31:43.903159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.903136    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2gv4s" event={"ID":"e4fd9469-a348-412a-abf5-c8ef100edb5e","Type":"ContainerStarted","Data":"6b41bf02fd4ea62353dfd3f19e61136601b38c78c167fcec44e84cb90cc56138"}
May 20 15:31:43.908774 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.908743    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g" event={"ID":"1a472476-846f-4b44-aa82-002fa5ca7715","Type":"ContainerStarted","Data":"07e1bd9a162a8d15b116779e7f4ea8a6eeca4ee21c7b0ca1e89460b4acf5e811"}
May 20 15:31:43.921737 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.921713    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wnpr5" event={"ID":"857b75bc-eba1-4597-a9bb-9855b531519c","Type":"ContainerStarted","Data":"403e9f7523c6c1381247516a7c801d2f37af8dd154f099258bf7bc66cbb877bf"}
May 20 15:31:43.927768 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.927745    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cjl8v" event={"ID":"f1736103-41b1-47b5-beac-d9442dd46cc0","Type":"ContainerStarted","Data":"cedb1f0662e7e762565587ec197f17b113e3c438640d89429f35ba0c15f7fb57"}
May 20 15:31:43.988794 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:43.988768    2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:44.252044 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.252017    2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:44.384889 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.384856    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:44.385061 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.385018    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:44.385153 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.385094    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:46.385060797 +0000 UTC m=+5.125811470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:44.485336 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.485301    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:44.485504 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.485489    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:44.485584 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.485512    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:44.485584 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.485525    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:44.485691 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.485587    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:46.485561419 +0000 UTC m=+5.226312088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:44.807950 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.807899    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:42 +0000 UTC" deadline="2027-12-11 10:39:09.090068526 +0000 UTC"
May 20 15:31:44.807950 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.807939    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13675h7m24.282133641s"
May 20 15:31:44.857825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:44.857798    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:44.857965 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:44.857924    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:45.859536 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:45.859507    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:45.859979 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:45.859631    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:46.403732 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:46.403696    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:46.403922 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.403833    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:46.403922 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.403891    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:50.403872122 +0000 UTC m=+9.144622784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:46.504676 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:46.504641    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:46.504859 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.504844    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:46.504920 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.504865    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:46.504920 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.504878    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:46.505022 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.504940    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:50.504920707 +0000 UTC m=+9.245671385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:46.857865 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:46.857837    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:46.858033 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:46.857973    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:47.857882 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:47.857849    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:47.858315 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:47.857976    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:48.857363 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:48.857329    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:48.857531 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:48.857498    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:49.857441 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:49.857386    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:49.857900 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:49.857509    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:50.437042 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:50.437009    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:50.437246 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.437169    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:50.437313 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.437251    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:58.437229117 +0000 UTC m=+17.177979787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:50.538345 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:50.538287    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:50.538570 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.538549    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:50.538628 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.538579    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:50.538628 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.538593    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:50.538721 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.538648    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:58.5386302 +0000 UTC m=+17.279380859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:50.858148 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:50.858000    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:50.858595 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:50.858155    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:51.273494 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.273461    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-g8b9j"]
May 20 15:31:51.278120 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.278098    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.278233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.278183    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:31:51.344075 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.344041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-dbus\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.344222 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.344105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-kubelet-config\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.344222 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.344174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.445382    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-kubelet-config\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.445446    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.445513    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-dbus\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.445519    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-kubelet-config\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.445644    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.445672    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9696679-eb3e-4d6a-b899-a87da586bd90-dbus\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.445764 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.445702    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:51.945682718 +0000 UTC m=+10.686433374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:51.858230 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.858199    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:51.858662 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.858302    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:51.949972 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:51.949941    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:51.950156 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.950115    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:51.950207 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:51.950188    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:52.950168463 +0000 UTC m=+11.690919120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:52.857336 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:52.857285    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:52.857527 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:52.857293    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:52.857527 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:52.857411    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:31:52.857659 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:52.857538    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:52.957577 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:52.957540    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:52.957929 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:52.957684    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:52.957929 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:52.957742    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:54.957725026 +0000 UTC m=+13.698475692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:53.857304 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:53.857269    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:53.857478 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:53.857396    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:54.857449 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:54.857420    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:54.857890 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:54.857424    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:54.857890 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:54.857526    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:31:54.857890 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:54.857643    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:54.974233 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:54.974195    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:54.974390 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:54.974332    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:54.974435 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:54.974394    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:58.974376558 +0000 UTC m=+17.715127216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:55.858018 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:55.857990    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:55.858403 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:55.858127    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:56.858246 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:56.858208    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:56.858695 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:56.858208    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:56.858695 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:56.858334    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:31:56.858695 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:56.858438    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:57.859660 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:57.859635    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:57.860028 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:57.859728    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:31:58.503313 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:58.503278    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:58.503579 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.503434    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:58.503579 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.503510    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:14.503487705 +0000 UTC m=+33.244238362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:58.604182 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:58.604146    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:58.604336 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.604304    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:58.604336 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.604324    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:58.604336 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.604337    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:58.604458 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.604392    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:14.604374172 +0000 UTC m=+33.345124831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:58.857188 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:58.857157    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:31:58.857365 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:58.857202    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:58.857365 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.857309    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:31:58.857467 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:58.857432    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:31:59.007592 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:59.007555    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:31:59.008028 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:59.007706    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:31:59.008028 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:59.007773    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:07.007752509 +0000 UTC m=+25.748503177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:31:59.857453 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:31:59.857422    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:31:59.857624 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:31:59.857540    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:00.857450 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:00.857418    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:00.857927 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:00.857423    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:00.857927 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:00.857536    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:00.857927 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:00.857637    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:01.859772 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.859548    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:01.859772 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:01.859673    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:01.955917 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.955887    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"6098c26b9e8566c02b46e171f4d6e27c4ff1d141b78fb08e0636da00b681cac1"}
May 20 15:32:01.956027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.955921    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"8e8028646f040e70ca96d719a95400562c9f5ffa88fc75696278fc7201bdc1a9"}
May 20 15:32:01.956027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.955941    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"43fc92c78c64357f7f697db1fc198abe93d8fdce8cd80388db61dd381098ac2e"}
May 20 15:32:01.956027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.955958    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"27f90f60c9822f3c42767773bdd917c0a2c136578efaf02ca6a5b21331d7ae98"}
May 20 15:32:01.958106 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.958059    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"a5244faf83b4a2df37ec043783dad277","Type":"ContainerStarted","Data":"3dc57e96c6d6362d5d2d32fefdbefa16799d222fa632d07aa66a2508719a3b2f"}
May 20 15:32:01.960109 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.960035    2574 generic.go:358] "Generic (PLEG): container finished" podID="a3aec373d631fee7d033910fa781e961" containerID="df3fb65eb7b8ee5a9a638857f724fd471dacadf21482a4afd393a580cb08c2ed" exitCode=0
May 20 15:32:01.960213 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.960112    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"a3aec373d631fee7d033910fa781e961","Type":"ContainerDied","Data":"df3fb65eb7b8ee5a9a638857f724fd471dacadf21482a4afd393a580cb08c2ed"}
May 20 15:32:01.961672 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.961647    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g" event={"ID":"1a472476-846f-4b44-aa82-002fa5ca7715","Type":"ContainerStarted","Data":"9b80d2305bee54fd0cdb71a3ac188a16cbd8c88ed6da67a88fc7e421df07cbd2"}
May 20 15:32:01.963007 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.962978    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wnpr5" event={"ID":"857b75bc-eba1-4597-a9bb-9855b531519c","Type":"ContainerStarted","Data":"efcee5b5a9970a0c9e6acef72d8654c40ad78b9146338446d1d3985cad7089c0"}
May 20 15:32:01.964400 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.964378    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cjl8v" event={"ID":"f1736103-41b1-47b5-beac-d9442dd46cc0","Type":"ContainerStarted","Data":"9be3ed4179df37821409cc2e0b4530ac7af71e393af7e8c15820c587a3331f7b"}
May 20 15:32:01.966452 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.966371    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerStarted","Data":"10c050e189f8ed511e4c956712364a8adfaa67705612f6d852c2e5bc4cb5be6e"}
May 20 15:32:01.967623 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.967600    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2rvk" event={"ID":"e4d43c9f-a1da-4bb9-afe8-ad383db25c79","Type":"ContainerStarted","Data":"bf16c72f4ead43c694bd1ec8cfc55e16d8b85da03efea7f886f05dfb0d86ce75"}
May 20 15:32:01.969540 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.969521    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdjgt" event={"ID":"c2edd435-0b32-47a9-91ef-48736be5945b","Type":"ContainerStarted","Data":"843e5f86540108e8ddc44e530509af8e5ba9ec13ae6be3cfe3ac31e33357f22b"}
May 20 15:32:01.971775 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.971742    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" podStartSLOduration=20.97171819 podStartE2EDuration="20.97171819s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:01.971590958 +0000 UTC m=+20.712341635" watchObservedRunningTime="2026-05-20 15:32:01.97171819 +0000 UTC m=+20.712468846"
May 20 15:32:01.985829 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:01.985719    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cjl8v" podStartSLOduration=2.671503589 podStartE2EDuration="20.985705564s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.106713024 +0000 UTC m=+1.847463680" lastFinishedPulling="2026-05-20 15:32:01.420914985 +0000 UTC m=+20.161665655" observedRunningTime="2026-05-20 15:32:01.98555177 +0000 UTC m=+20.726302448" watchObservedRunningTime="2026-05-20 15:32:01.985705564 +0000 UTC m=+20.726456243"
May 20 15:32:02.001376 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.001331    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wnpr5" podStartSLOduration=1.668107217 podStartE2EDuration="20.001317921s" podCreationTimestamp="2026-05-20 15:31:42 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.110825025 +0000 UTC m=+1.851575681" lastFinishedPulling="2026-05-20 15:32:01.444035729 +0000 UTC m=+20.184786385" observedRunningTime="2026-05-20 15:32:02.001306686 +0000 UTC m=+20.742057365" watchObservedRunningTime="2026-05-20 15:32:02.001317921 +0000 UTC m=+20.742068599"
May 20 15:32:02.054167 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.054122    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-p2rvk" podStartSLOduration=2.678252511 podStartE2EDuration="21.054104241s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.101244472 +0000 UTC m=+1.841995128" lastFinishedPulling="2026-05-20 15:32:01.477096199 +0000 UTC m=+20.217846858" observedRunningTime="2026-05-20 15:32:02.05381267 +0000 UTC m=+20.794563348" watchObservedRunningTime="2026-05-20 15:32:02.054104241 +0000 UTC m=+20.794854914"
May 20 15:32:02.069225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.069185    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gdjgt" podStartSLOduration=6.651438767 podStartE2EDuration="21.069174091s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.096796343 +0000 UTC m=+1.837547011" lastFinishedPulling="2026-05-20 15:31:57.514531678 +0000 UTC m=+16.255282335" observedRunningTime="2026-05-20 15:32:02.068847984 +0000 UTC m=+20.809598665" watchObservedRunningTime="2026-05-20 15:32:02.069174091 +0000 UTC m=+20.809924769"
May 20 15:32:02.733870 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.733840    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 15:32:02.833854 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.833776    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T15:32:02.733867136Z","UUID":"a75098ec-ba82-4858-8dea-221e67f48c0e","Handler":null,"Name":"","Endpoint":""}
May 20 15:32:02.835425 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.835407    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 15:32:02.835425 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.835430    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 15:32:02.857167 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.857149    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:02.857240 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.857192    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:02.857292 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:02.857273    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:02.857346 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:02.857332    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:02.972231 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.972197    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"a3aec373d631fee7d033910fa781e961","Type":"ContainerStarted","Data":"e22776ffff791f9e1f049055bb7d4ba7a8d4b8c53fad6c49addd0a64a8dc095b"}
May 20 15:32:02.973608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.973585    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g" event={"ID":"1a472476-846f-4b44-aa82-002fa5ca7715","Type":"ContainerStarted","Data":"44600ff1e99d50a93bf0ad38e89024cc5c30a0e0d78f655f1a0d50cb27e23a91"}
May 20 15:32:02.974803 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.974779    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="10c050e189f8ed511e4c956712364a8adfaa67705612f6d852c2e5bc4cb5be6e" exitCode=0
May 20 15:32:02.974883 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.974856    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"10c050e189f8ed511e4c956712364a8adfaa67705612f6d852c2e5bc4cb5be6e"}
May 20 15:32:02.977512 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.977433    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"97f4621016e101902b24a64de9a3fa358928a0fab4791d2df7925af46531ea08"}
May 20 15:32:02.977512 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.977457    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"28d2715df5835dfd27b4392b39e2017dce7eb755137d503fc5bb4f892ee3ab86"}
May 20 15:32:02.978746 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:02.978690    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2gv4s" event={"ID":"e4fd9469-a348-412a-abf5-c8ef100edb5e","Type":"ContainerStarted","Data":"41c772c3c3f5b214c18411475d33fdd55be7698110567ad8f01bb4ff7f7276f1"}
May 20 15:32:03.007963 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.007920    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" podStartSLOduration=22.007910066 podStartE2EDuration="22.007910066s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:02.991575752 +0000 UTC m=+21.732326429" watchObservedRunningTime="2026-05-20 15:32:03.007910066 +0000 UTC m=+21.748660744"
May 20 15:32:03.008108 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.008073    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2gv4s" podStartSLOduration=3.631710059 podStartE2EDuration="22.008069174s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.066042312 +0000 UTC m=+1.806792968" lastFinishedPulling="2026-05-20 15:32:01.442401414 +0000 UTC m=+20.183152083" observedRunningTime="2026-05-20 15:32:03.007884173 +0000 UTC m=+21.748634861" watchObservedRunningTime="2026-05-20 15:32:03.008069174 +0000 UTC m=+21.748819852"
May 20 15:32:03.759988 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.759908    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:32:03.760574 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.760550    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:32:03.857947 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.857917    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:03.858102 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:03.858037    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:03.981834 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.981805    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g" event={"ID":"1a472476-846f-4b44-aa82-002fa5ca7715","Type":"ContainerStarted","Data":"aaea22faa4b23d34ffb9082eeab3b64f5bf2e70b6a4930fc698fa8498160adb7"}
May 20 15:32:03.982545 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.982522    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:32:03.982910 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:03.982893    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gdjgt"
May 20 15:32:04.001252 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:04.001212    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2t8g" podStartSLOduration=1.627176247 podStartE2EDuration="22.001200853s" podCreationTimestamp="2026-05-20 15:31:42 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.117250446 +0000 UTC m=+1.858001107" lastFinishedPulling="2026-05-20 15:32:03.491275054 +0000 UTC m=+22.232025713" observedRunningTime="2026-05-20 15:32:04.000887523 +0000 UTC m=+22.741638220" watchObservedRunningTime="2026-05-20 15:32:04.001200853 +0000 UTC m=+22.741951530"
May 20 15:32:04.857521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:04.857361    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:04.857685 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:04.857606    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:04.857685 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:04.857358    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:04.857764 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:04.857709    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:04.986863 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:04.986825    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"c589a012f30c6d63aaa9e588f636a1030e2350ca7d70973890e62cf1c5b16552"}
May 20 15:32:05.375062 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.375026    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-747vx"]
May 20 15:32:05.379766 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.379736    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.382762 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.382732    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w9tn4\""
May 20 15:32:05.382968 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.382940    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 15:32:05.383526 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.383510    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 15:32:05.454736 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.454713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d325a410-f630-445c-a538-feddddf807d2-tmp-dir\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.454873 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.454747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zstl\" (UniqueName: \"kubernetes.io/projected/d325a410-f630-445c-a538-feddddf807d2-kube-api-access-6zstl\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.454873 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.454773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d325a410-f630-445c-a538-feddddf807d2-hosts-file\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.555979 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.555947    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d325a410-f630-445c-a538-feddddf807d2-tmp-dir\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.556160 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.555999    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6zstl\" (UniqueName: \"kubernetes.io/projected/d325a410-f630-445c-a538-feddddf807d2-kube-api-access-6zstl\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.556160 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.556032    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d325a410-f630-445c-a538-feddddf807d2-hosts-file\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.556276 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.556181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d325a410-f630-445c-a538-feddddf807d2-hosts-file\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.556361 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.556341    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d325a410-f630-445c-a538-feddddf807d2-tmp-dir\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.567304 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.567284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zstl\" (UniqueName: \"kubernetes.io/projected/d325a410-f630-445c-a538-feddddf807d2-kube-api-access-6zstl\") pod \"node-resolver-747vx\" (UID: \"d325a410-f630-445c-a538-feddddf807d2\") " pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.690571 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.690507    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-747vx"
May 20 15:32:05.857789 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:05.857756    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:05.857944 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:05.857886    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:06.857850 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:06.857823    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:06.857850 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:06.857841    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:06.858456 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:06.857944    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:06.858456 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:06.858059    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:07.055793 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:07.055767    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd325a410_f630_445c_a538_feddddf807d2.slice/crio-18eb9239a0f2ef8342c19f92d57c03571b22d01f1cd6ddf8f7f85ba3d0c0c058 WatchSource:0}: Error finding container 18eb9239a0f2ef8342c19f92d57c03571b22d01f1cd6ddf8f7f85ba3d0c0c058: Status 404 returned error can't find the container with id 18eb9239a0f2ef8342c19f92d57c03571b22d01f1cd6ddf8f7f85ba3d0c0c058
May 20 15:32:07.070933 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.070911    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:07.071060 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:07.071044    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 15:32:07.071120 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:07.071112    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret podName:f9696679-eb3e-4d6a-b899-a87da586bd90 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:23.071094179 +0000 UTC m=+41.811844851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret") pod "global-pull-secret-syncer-g8b9j" (UID: "f9696679-eb3e-4d6a-b899-a87da586bd90") : object "kube-system"/"original-pull-secret" not registered
May 20 15:32:07.858346 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.858185    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:07.858882 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:07.858405    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:07.994733 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.994702    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="1fc172c35a5929c3fca7328528c6d8a4cb365e0871cf4e33458f2f722dca4bd7" exitCode=0
May 20 15:32:07.994955 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.994743    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"1fc172c35a5929c3fca7328528c6d8a4cb365e0871cf4e33458f2f722dca4bd7"}
May 20 15:32:07.997975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.997952    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" event={"ID":"4c34c181-6934-4c86-bf7d-71221abd751c","Type":"ContainerStarted","Data":"86dd218bcdf0ec4714c66ecb0b3bacd46be69190d83b4153ae4b8c9eae19361e"}
May 20 15:32:07.998292 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.998245    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:07.998292 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.998274    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:07.999415 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.999389    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-747vx" event={"ID":"d325a410-f630-445c-a538-feddddf807d2","Type":"ContainerStarted","Data":"bc5b0fdfcefafa3733a4b252a18bd827a9026d0e39c295fa82609d263c3976fe"}
May 20 15:32:07.999509 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:07.999425    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-747vx" event={"ID":"d325a410-f630-445c-a538-feddddf807d2","Type":"ContainerStarted","Data":"18eb9239a0f2ef8342c19f92d57c03571b22d01f1cd6ddf8f7f85ba3d0c0c058"}
May 20 15:32:08.013093 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:08.013068    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:08.034608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:08.034573    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-747vx" podStartSLOduration=3.034562261 podStartE2EDuration="3.034562261s" podCreationTimestamp="2026-05-20 15:32:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:08.034213925 +0000 UTC m=+26.774964615" watchObservedRunningTime="2026-05-20 15:32:08.034562261 +0000 UTC m=+26.775312939"
May 20 15:32:08.064553 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:08.064516    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd" podStartSLOduration=8.540410326 podStartE2EDuration="27.064505889s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.080935028 +0000 UTC m=+1.821685683" lastFinishedPulling="2026-05-20 15:32:01.605030589 +0000 UTC m=+20.345781246" observedRunningTime="2026-05-20 15:32:08.06374041 +0000 UTC m=+26.804491088" watchObservedRunningTime="2026-05-20 15:32:08.064505889 +0000 UTC m=+26.805256564"
May 20 15:32:08.857899 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:08.857828    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:08.858050 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:08.857830    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:08.858050 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:08.857948    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:08.858050 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:08.857997    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:09.002697 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.002668    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="b3a300069e20c42e83dd126c8c491c72923efc90985f3ad852f08420af2c551a" exitCode=0
May 20 15:32:09.003016 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.002739    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"b3a300069e20c42e83dd126c8c491c72923efc90985f3ad852f08420af2c551a"}
May 20 15:32:09.003196 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.003178    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:09.017895 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.017876    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:09.348939 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.348911    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g8b9j"]
May 20 15:32:09.349128 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.349039    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:09.349201 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:09.349158    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:09.351380 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.351352    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qmkpv"]
May 20 15:32:09.351506 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.351458    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:09.351568 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:09.351545    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:09.355518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.355494    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dz424"]
May 20 15:32:09.355645 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:09.355602    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:09.355726 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:09.355702    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:10.006725 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:10.006553    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="1b2d30a0115c8dbe3fcc5fa8ad6632754859093d98970d15f69438965e8f080c" exitCode=0
May 20 15:32:10.007164 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:10.006638    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"1b2d30a0115c8dbe3fcc5fa8ad6632754859093d98970d15f69438965e8f080c"}
May 20 15:32:10.857642 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:10.857611    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:10.857817 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:10.857611    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:10.857817 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:10.857742    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:10.857916 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:10.857817    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:10.857916 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:10.857617    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:10.858020 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:10.857916    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:12.857923 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:12.857886    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:12.858636 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:12.858024    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qmkpv" podUID="9e4f292c-23eb-45d1-be14-7e368f49ff38"
May 20 15:32:12.858636 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:12.857886    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:12.858636 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:12.858149    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g8b9j" podUID="f9696679-eb3e-4d6a-b899-a87da586bd90"
May 20 15:32:12.858636 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:12.857896    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:12.858636 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:12.858227    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dz424" podUID="64cd1e6e-498b-43df-9ebc-cdc2771a6ad2"
May 20 15:32:14.117782 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.117736    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeReady"
May 20 15:32:14.118212 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.117878    2574 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 15:32:14.159294 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.159267    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:32:14.182351 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.182326    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:32:14.182480 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.182361    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.186561 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.186538    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 15:32:14.187311 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.186538    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 15:32:14.187616 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.187585    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 15:32:14.187720 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.187633    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-62gbd\""
May 20 15:32:14.187810 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.187742    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6zcsn"]
May 20 15:32:14.197802 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.197786    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 15:32:14.202225 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.202208    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-22rt8"]
May 20 15:32:14.202372 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.202346    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.205844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.205826    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 15:32:14.205943 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.205859    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 15:32:14.205943 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.205900    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 15:32:14.206028 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.205976    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5s49c\""
May 20 15:32:14.220443 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.220426    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zcsn"]
May 20 15:32:14.220581 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.220565    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.226688 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.226504    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 15:32:14.226688 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.226511    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zngvt\""
May 20 15:32:14.227053 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.227037    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-22rt8"]
May 20 15:32:14.227610 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.227595    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 15:32:14.323032 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.322960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323032 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcfk\" (UniqueName: \"kubernetes.io/projected/e5078f38-1131-44ce-8880-29fe9ccfd946-kube-api-access-4dcfk\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.323254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9cx\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgmq\" (UniqueName: \"kubernetes.io/projected/29045f79-cc96-419a-8c83-dc73b9bd3bf0-kube-api-access-hsgmq\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.323254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29045f79-cc96-419a-8c83-dc73b9bd3bf0-config-volume\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.323254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.323254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29045f79-cc96-419a-8c83-dc73b9bd3bf0-tmp-dir\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.323498 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.323715 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.323511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.424819 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.424785    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9cx\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425020 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.424844    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgmq\" (UniqueName: \"kubernetes.io/projected/29045f79-cc96-419a-8c83-dc73b9bd3bf0-kube-api-access-hsgmq\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.425020 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.424874    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29045f79-cc96-419a-8c83-dc73b9bd3bf0-config-volume\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.425020 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.424899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.425020 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425008    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:14.425251 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425075    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls podName:29045f79-cc96-419a-8c83-dc73b9bd3bf0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:14.925053708 +0000 UTC m=+33.665804369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls") pod "dns-default-22rt8" (UID: "29045f79-cc96-419a-8c83-dc73b9bd3bf0") : secret "dns-default-metrics-tls" not found
May 20 15:32:14.425251 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425145    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425251 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425186    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425251 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425216    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425251 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425234    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425253    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29045f79-cc96-419a-8c83-dc73b9bd3bf0-tmp-dir\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425279    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425296    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425313    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425331    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425343    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85768dcc7d-f5rdm: secret "image-registry-tls" not found
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425383    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls podName:d852f365-9ffe-4dd9-a72b-dfe53d974547 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:14.925367197 +0000 UTC m=+33.666117911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls") pod "image-registry-85768dcc7d-f5rdm" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547") : secret "image-registry-tls" not found
May 20 15:32:14.425505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425464    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29045f79-cc96-419a-8c83-dc73b9bd3bf0-config-volume\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.425914 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425551    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:14.425914 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.425609    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert podName:e5078f38-1131-44ce-8880-29fe9ccfd946 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:14.925594581 +0000 UTC m=+33.666345237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert") pod "ingress-canary-6zcsn" (UID: "e5078f38-1131-44ce-8880-29fe9ccfd946") : secret "canary-serving-cert" not found
May 20 15:32:14.425914 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425692    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.426017 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425937    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.426017 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425331    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.426017 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.425989    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcfk\" (UniqueName: \"kubernetes.io/projected/e5078f38-1131-44ce-8880-29fe9ccfd946-kube-api-access-4dcfk\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.426149 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.426102    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29045f79-cc96-419a-8c83-dc73b9bd3bf0-tmp-dir\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.426949 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.426909    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.430048 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.429957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.430048 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.429973    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.439650 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.439621    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9cx\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.440649 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.440624    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgmq\" (UniqueName: \"kubernetes.io/projected/29045f79-cc96-419a-8c83-dc73b9bd3bf0-kube-api-access-hsgmq\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.440781 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.440762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.441053 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.441037    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcfk\" (UniqueName: \"kubernetes.io/projected/e5078f38-1131-44ce-8880-29fe9ccfd946-kube-api-access-4dcfk\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.526927 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.526898    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:14.527096 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.527053    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:14.527197 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.527145    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs podName:9e4f292c-23eb-45d1-be14-7e368f49ff38 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:46.527125184 +0000 UTC m=+65.267875847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs") pod "network-metrics-daemon-qmkpv" (UID: "9e4f292c-23eb-45d1-be14-7e368f49ff38") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:14.627424 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.627382    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:14.627581 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.627553    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:14.627581 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.627576    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:14.627695 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.627591    2574 projected.go:194] Error preparing data for projected volume kube-api-access-56zf8 for pod openshift-network-diagnostics/network-check-target-dz424: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:14.627695 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.627667    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8 podName:64cd1e6e-498b-43df-9ebc-cdc2771a6ad2 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:46.627649453 +0000 UTC m=+65.368400126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-56zf8" (UniqueName: "kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8") pod "network-check-target-dz424" (UID: "64cd1e6e-498b-43df-9ebc-cdc2771a6ad2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:14.857623 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.857553    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:14.857623 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.857597    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:14.857881 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.857570    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:14.860350 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860328    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:14.860350 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860338    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:14.860521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860383    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 15:32:14.860521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860402    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9zz48\""
May 20 15:32:14.860521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860434    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:14.860724 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.860708    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm9j9\""
May 20 15:32:14.929916 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.929885    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:14.930050 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.929932    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:14.930050 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:14.929988    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:14.930050 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930038    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930058    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85768dcc7d-f5rdm: secret "image-registry-tls" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930124    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930131    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930136    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls podName:d852f365-9ffe-4dd9-a72b-dfe53d974547 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:15.930118474 +0000 UTC m=+34.670869131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls") pod "image-registry-85768dcc7d-f5rdm" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547") : secret "image-registry-tls" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930211    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert podName:e5078f38-1131-44ce-8880-29fe9ccfd946 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:15.930192531 +0000 UTC m=+34.670943188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert") pod "ingress-canary-6zcsn" (UID: "e5078f38-1131-44ce-8880-29fe9ccfd946") : secret "canary-serving-cert" not found
May 20 15:32:14.930233 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:14.930229    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls podName:29045f79-cc96-419a-8c83-dc73b9bd3bf0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:15.930218577 +0000 UTC m=+34.670969241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls") pod "dns-default-22rt8" (UID: "29045f79-cc96-419a-8c83-dc73b9bd3bf0") : secret "dns-default-metrics-tls" not found
May 20 15:32:15.938415 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:15.938387    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:15.938470    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:15.938513    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938560    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938630    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls podName:29045f79-cc96-419a-8c83-dc73b9bd3bf0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:17.938611148 +0000 UTC m=+36.679361822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls") pod "dns-default-22rt8" (UID: "29045f79-cc96-419a-8c83-dc73b9bd3bf0") : secret "dns-default-metrics-tls" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938634    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938642    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938648    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85768dcc7d-f5rdm: secret "image-registry-tls" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938681    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls podName:d852f365-9ffe-4dd9-a72b-dfe53d974547 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:17.938670693 +0000 UTC m=+36.679421360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls") pod "image-registry-85768dcc7d-f5rdm" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547") : secret "image-registry-tls" not found
May 20 15:32:15.938905 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:15.938696    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert podName:e5078f38-1131-44ce-8880-29fe9ccfd946 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:17.938688017 +0000 UTC m=+36.679438674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert") pod "ingress-canary-6zcsn" (UID: "e5078f38-1131-44ce-8880-29fe9ccfd946") : secret "canary-serving-cert" not found
May 20 15:32:15.983311 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:15.983167    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"]
May 20 15:32:16.015158 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.015140    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"]
May 20 15:32:16.015259 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.015242    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"
May 20 15:32:16.021096 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.021059    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dvdpm\""
May 20 15:32:16.021949 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.021929    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 20 15:32:16.021949 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.021941    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 20 15:32:16.139739 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.139562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjfm\" (UniqueName: \"kubernetes.io/projected/bb60b51c-622c-4185-a4fa-8b894b7b1de3-kube-api-access-vhjfm\") pod \"migrator-b985f5d89-9drjl\" (UID: \"bb60b51c-622c-4185-a4fa-8b894b7b1de3\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"
May 20 15:32:16.240851 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.240828    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjfm\" (UniqueName: \"kubernetes.io/projected/bb60b51c-622c-4185-a4fa-8b894b7b1de3-kube-api-access-vhjfm\") pod \"migrator-b985f5d89-9drjl\" (UID: \"bb60b51c-622c-4185-a4fa-8b894b7b1de3\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"
May 20 15:32:16.250419 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.250399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjfm\" (UniqueName: \"kubernetes.io/projected/bb60b51c-622c-4185-a4fa-8b894b7b1de3-kube-api-access-vhjfm\") pod \"migrator-b985f5d89-9drjl\" (UID: \"bb60b51c-622c-4185-a4fa-8b894b7b1de3\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"
May 20 15:32:16.323647 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.323627    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"
May 20 15:32:16.509214 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:16.509151    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl"]
May 20 15:32:16.513356 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:16.513335    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb60b51c_622c_4185_a4fa_8b894b7b1de3.slice/crio-c6b8da78316ec1bdfd718458159c7c9e0c4876c10464d8822358fced77bb42f4 WatchSource:0}: Error finding container c6b8da78316ec1bdfd718458159c7c9e0c4876c10464d8822358fced77bb42f4: Status 404 returned error can't find the container with id c6b8da78316ec1bdfd718458159c7c9e0c4876c10464d8822358fced77bb42f4
May 20 15:32:17.018929 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.018890    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl" event={"ID":"bb60b51c-622c-4185-a4fa-8b894b7b1de3","Type":"ContainerStarted","Data":"c6b8da78316ec1bdfd718458159c7c9e0c4876c10464d8822358fced77bb42f4"}
May 20 15:32:17.021137 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.021117    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="50d06d343797fd618ffb9d18c262a868826c001c8d76a1a5d2fb21747c580534" exitCode=0
May 20 15:32:17.021230 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.021165    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"50d06d343797fd618ffb9d18c262a868826c001c8d76a1a5d2fb21747c580534"}
May 20 15:32:17.207885 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.207862    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-747vx_d325a410-f630-445c-a538-feddddf807d2/dns-node-resolver/0.log"
May 20 15:32:17.955729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.955699    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:17.955729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.955735    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:17.955913 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955850    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:17.955913 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955872    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85768dcc7d-f5rdm: secret "image-registry-tls" not found
May 20 15:32:17.955913 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:17.955872    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:17.956018 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955928    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls podName:d852f365-9ffe-4dd9-a72b-dfe53d974547 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.955910597 +0000 UTC m=+40.696661274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls") pod "image-registry-85768dcc7d-f5rdm" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547") : secret "image-registry-tls" not found
May 20 15:32:17.956018 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955959    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:17.956018 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955968    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:17.956018 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.955995    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert podName:e5078f38-1131-44ce-8880-29fe9ccfd946 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.955984674 +0000 UTC m=+40.696735330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert") pod "ingress-canary-6zcsn" (UID: "e5078f38-1131-44ce-8880-29fe9ccfd946") : secret "canary-serving-cert" not found
May 20 15:32:17.956018 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:17.956008    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls podName:29045f79-cc96-419a-8c83-dc73b9bd3bf0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:21.956001792 +0000 UTC m=+40.696752448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls") pod "dns-default-22rt8" (UID: "29045f79-cc96-419a-8c83-dc73b9bd3bf0") : secret "dns-default-metrics-tls" not found
May 20 15:32:18.025986 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.025819    2574 generic.go:358] "Generic (PLEG): container finished" podID="43619788-ee3e-4703-a24c-29887c78e05f" containerID="c17d74fef7a9928b5058a4fab4bedc45bd78a3dcd1a3d36b97b67664ca451571" exitCode=0
May 20 15:32:18.026345 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.025904    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerDied","Data":"c17d74fef7a9928b5058a4fab4bedc45bd78a3dcd1a3d36b97b67664ca451571"}
May 20 15:32:18.406465 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.406441    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cjl8v_f1736103-41b1-47b5-beac-d9442dd46cc0/node-ca/0.log"
May 20 15:32:18.856280 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.856249    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-4qqcp"]
May 20 15:32:18.878531 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.878503    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-4qqcp"]
May 20 15:32:18.878672 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.878610    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:18.884272 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.884251    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 20 15:32:18.884272 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.884268    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 20 15:32:18.884430 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.884251    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n6mrm\""
May 20 15:32:18.884685 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.884668    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 20 15:32:18.886674 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:18.886656    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 20 15:32:19.029931 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.029865    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl" event={"ID":"bb60b51c-622c-4185-a4fa-8b894b7b1de3","Type":"ContainerStarted","Data":"56b1f1da872eaadb29f0d1a1823835198d540be151485bc0a3d79ad889bf94ab"}
May 20 15:32:19.029931 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.029896    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl" event={"ID":"bb60b51c-622c-4185-a4fa-8b894b7b1de3","Type":"ContainerStarted","Data":"9ea8bdf6c6a80b8dcca46bad6af2f461973e98ab26e5a0baf4442743b22c3869"}
May 20 15:32:19.032661 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.032639    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" event={"ID":"43619788-ee3e-4703-a24c-29887c78e05f","Type":"ContainerStarted","Data":"c784e465308da57667da23efcbe082aa0c2609c4e0691556cf927ee9cdcb348e"}
May 20 15:32:19.060575 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.060537    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-9drjl" podStartSLOduration=2.605498573 podStartE2EDuration="4.060526267s" podCreationTimestamp="2026-05-20 15:32:15 +0000 UTC" firstStartedPulling="2026-05-20 15:32:16.515068697 +0000 UTC m=+35.255819352" lastFinishedPulling="2026-05-20 15:32:17.970096374 +0000 UTC m=+36.710847046" observedRunningTime="2026-05-20 15:32:19.05998249 +0000 UTC m=+37.800733168" watchObservedRunningTime="2026-05-20 15:32:19.060526267 +0000 UTC m=+37.801276949"
May 20 15:32:19.065230 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.065211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xhr\" (UniqueName: \"kubernetes.io/projected/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-kube-api-access-95xhr\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.065316 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.065246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-key\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.065316 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.065263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-cabundle\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.113769 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.113737    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ct8vp" podStartSLOduration=4.335538819 podStartE2EDuration="37.113726609s" podCreationTimestamp="2026-05-20 15:31:42 +0000 UTC" firstStartedPulling="2026-05-20 15:31:43.122958445 +0000 UTC m=+1.863709117" lastFinishedPulling="2026-05-20 15:32:15.901146246 +0000 UTC m=+34.641896907" observedRunningTime="2026-05-20 15:32:19.113676286 +0000 UTC m=+37.854426961" watchObservedRunningTime="2026-05-20 15:32:19.113726609 +0000 UTC m=+37.854477279"
May 20 15:32:19.165852 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.165832    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-95xhr\" (UniqueName: \"kubernetes.io/projected/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-kube-api-access-95xhr\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.165952 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.165863    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-key\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.165952 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.165879    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-cabundle\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.166528 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.166497    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-cabundle\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.169630 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.169611    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-signing-key\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.175689 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.175664    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xhr\" (UniqueName: \"kubernetes.io/projected/8d19ce43-04c9-4a96-a7c4-cea32bbf17b6-kube-api-access-95xhr\") pod \"service-ca-67d667c7d-4qqcp\" (UID: \"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6\") " pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.187133 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.187115    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-4qqcp"
May 20 15:32:19.360167 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:19.360141    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-4qqcp"]
May 20 15:32:19.366010 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:19.365980    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d19ce43_04c9_4a96_a7c4_cea32bbf17b6.slice/crio-86793630a335c27bb2444b259bf5387abc0ef787bad950538b75cee73dde3ccb WatchSource:0}: Error finding container 86793630a335c27bb2444b259bf5387abc0ef787bad950538b75cee73dde3ccb: Status 404 returned error can't find the container with id 86793630a335c27bb2444b259bf5387abc0ef787bad950538b75cee73dde3ccb
May 20 15:32:20.035937 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:20.035895    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-4qqcp" event={"ID":"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6","Type":"ContainerStarted","Data":"86793630a335c27bb2444b259bf5387abc0ef787bad950538b75cee73dde3ccb"}
May 20 15:32:21.987349 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:21.987115    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:21.987374    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987252    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987427    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85768dcc7d-f5rdm: secret "image-registry-tls" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:21.987449    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987475    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls podName:d852f365-9ffe-4dd9-a72b-dfe53d974547 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:29.98746039 +0000 UTC m=+48.728211047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls") pod "image-registry-85768dcc7d-f5rdm" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547") : secret "image-registry-tls" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987535    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987575    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987590    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls podName:29045f79-cc96-419a-8c83-dc73b9bd3bf0 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:29.987575922 +0000 UTC m=+48.728326589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls") pod "dns-default-22rt8" (UID: "29045f79-cc96-419a-8c83-dc73b9bd3bf0") : secret "dns-default-metrics-tls" not found
May 20 15:32:21.987762 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:21.987632    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert podName:e5078f38-1131-44ce-8880-29fe9ccfd946 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:29.987614443 +0000 UTC m=+48.728365115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert") pod "ingress-canary-6zcsn" (UID: "e5078f38-1131-44ce-8880-29fe9ccfd946") : secret "canary-serving-cert" not found
May 20 15:32:22.041488 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:22.041461    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-4qqcp" event={"ID":"8d19ce43-04c9-4a96-a7c4-cea32bbf17b6","Type":"ContainerStarted","Data":"12402da1937bde679629ba2126457f29707665a0ea83aa10de218003c62f4b51"}
May 20 15:32:22.063457 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:22.063415    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-4qqcp" podStartSLOduration=1.890385172 podStartE2EDuration="4.063400837s" podCreationTimestamp="2026-05-20 15:32:18 +0000 UTC" firstStartedPulling="2026-05-20 15:32:19.367883018 +0000 UTC m=+38.108633675" lastFinishedPulling="2026-05-20 15:32:21.540898679 +0000 UTC m=+40.281649340" observedRunningTime="2026-05-20 15:32:22.062236689 +0000 UTC m=+40.802987364" watchObservedRunningTime="2026-05-20 15:32:22.063400837 +0000 UTC m=+40.804151515"
May 20 15:32:23.095506 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:23.095477    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:23.098239 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:23.098212    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9696679-eb3e-4d6a-b899-a87da586bd90-original-pull-secret\") pod \"global-pull-secret-syncer-g8b9j\" (UID: \"f9696679-eb3e-4d6a-b899-a87da586bd90\") " pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:23.281331 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:23.281297    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g8b9j"
May 20 15:32:23.400344 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:23.400311    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g8b9j"]
May 20 15:32:23.403553 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:23.403522    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9696679_eb3e_4d6a_b899_a87da586bd90.slice/crio-48eda897ba162bf7fd2b80f4db1d365324c1626438c0f012d417c05e076c0e92 WatchSource:0}: Error finding container 48eda897ba162bf7fd2b80f4db1d365324c1626438c0f012d417c05e076c0e92: Status 404 returned error can't find the container with id 48eda897ba162bf7fd2b80f4db1d365324c1626438c0f012d417c05e076c0e92
May 20 15:32:24.046914 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:24.046876    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g8b9j" event={"ID":"f9696679-eb3e-4d6a-b899-a87da586bd90","Type":"ContainerStarted","Data":"48eda897ba162bf7fd2b80f4db1d365324c1626438c0f012d417c05e076c0e92"}
May 20 15:32:28.057517 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:28.057483    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g8b9j" event={"ID":"f9696679-eb3e-4d6a-b899-a87da586bd90","Type":"ContainerStarted","Data":"fefa29d0c997cba50a8462d211683de8837cd4688ae79e0527a9b64a0f361613"}
May 20 15:32:28.073574 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:28.073524    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g8b9j" podStartSLOduration=33.1074335 podStartE2EDuration="37.073510674s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:32:23.405358839 +0000 UTC m=+42.146109495" lastFinishedPulling="2026-05-20 15:32:27.371435999 +0000 UTC m=+46.112186669" observedRunningTime="2026-05-20 15:32:28.073106243 +0000 UTC m=+46.813856922" watchObservedRunningTime="2026-05-20 15:32:28.073510674 +0000 UTC m=+46.814261351"
May 20 15:32:30.056805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.056763    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:30.057362 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.056837    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:30.057362 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.056866    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:30.059240 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.059215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29045f79-cc96-419a-8c83-dc73b9bd3bf0-metrics-tls\") pod \"dns-default-22rt8\" (UID: \"29045f79-cc96-419a-8c83-dc73b9bd3bf0\") " pod="openshift-dns/dns-default-22rt8"
May 20 15:32:30.059727 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.059710    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5078f38-1131-44ce-8880-29fe9ccfd946-cert\") pod \"ingress-canary-6zcsn\" (UID: \"e5078f38-1131-44ce-8880-29fe9ccfd946\") " pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:30.059806 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.059736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"image-registry-85768dcc7d-f5rdm\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") " pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:30.094419 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.094395    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:30.111461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.111435    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zcsn"
May 20 15:32:30.130119 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.130078    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-22rt8"
May 20 15:32:30.256277 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.255856    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:32:30.260792 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:30.260763    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd852f365_9ffe_4dd9_a72b_dfe53d974547.slice/crio-fad661cc219f78296ce2163b3b45da8353cf62fe9e1f9b06dcc8dab3b473a3f0 WatchSource:0}: Error finding container fad661cc219f78296ce2163b3b45da8353cf62fe9e1f9b06dcc8dab3b473a3f0: Status 404 returned error can't find the container with id fad661cc219f78296ce2163b3b45da8353cf62fe9e1f9b06dcc8dab3b473a3f0
May 20 15:32:30.271960 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.271879    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zcsn"]
May 20 15:32:30.281254 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:30.276202    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5078f38_1131_44ce_8880_29fe9ccfd946.slice/crio-3ff71825345f1f4431e4c99498b761f652b2719ea35889d00bbd2e9907a364ab WatchSource:0}: Error finding container 3ff71825345f1f4431e4c99498b761f652b2719ea35889d00bbd2e9907a364ab: Status 404 returned error can't find the container with id 3ff71825345f1f4431e4c99498b761f652b2719ea35889d00bbd2e9907a364ab
May 20 15:32:30.308973 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:30.308921    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-22rt8"]
May 20 15:32:30.312918 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:30.312878    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29045f79_cc96_419a_8c83_dc73b9bd3bf0.slice/crio-08b7160de9b44768436d92f0c68b971052127f317be296e33d2da715a50b6ee4 WatchSource:0}: Error finding container 08b7160de9b44768436d92f0c68b971052127f317be296e33d2da715a50b6ee4: Status 404 returned error can't find the container with id 08b7160de9b44768436d92f0c68b971052127f317be296e33d2da715a50b6ee4
May 20 15:32:31.064487 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.064439    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zcsn" event={"ID":"e5078f38-1131-44ce-8880-29fe9ccfd946","Type":"ContainerStarted","Data":"3ff71825345f1f4431e4c99498b761f652b2719ea35889d00bbd2e9907a364ab"}
May 20 15:32:31.065665 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.065619    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22rt8" event={"ID":"29045f79-cc96-419a-8c83-dc73b9bd3bf0","Type":"ContainerStarted","Data":"08b7160de9b44768436d92f0c68b971052127f317be296e33d2da715a50b6ee4"}
May 20 15:32:31.067099 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.067057    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" event={"ID":"d852f365-9ffe-4dd9-a72b-dfe53d974547","Type":"ContainerStarted","Data":"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"}
May 20 15:32:31.067227 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.067103    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" event={"ID":"d852f365-9ffe-4dd9-a72b-dfe53d974547","Type":"ContainerStarted","Data":"fad661cc219f78296ce2163b3b45da8353cf62fe9e1f9b06dcc8dab3b473a3f0"}
May 20 15:32:31.067284 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.067253    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:31.878827 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:31.878774    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" podStartSLOduration=44.878642559 podStartE2EDuration="44.878642559s" podCreationTimestamp="2026-05-20 15:31:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:31.093168593 +0000 UTC m=+49.833919271" watchObservedRunningTime="2026-05-20 15:32:31.878642559 +0000 UTC m=+50.619393238"
May 20 15:32:33.075267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:33.075229    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22rt8" event={"ID":"29045f79-cc96-419a-8c83-dc73b9bd3bf0","Type":"ContainerStarted","Data":"ede2847b63b488c2025a50208414c035e1d6ad3c9d931260f40385f770d12ef2"}
May 20 15:32:33.075267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:33.075268    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22rt8" event={"ID":"29045f79-cc96-419a-8c83-dc73b9bd3bf0","Type":"ContainerStarted","Data":"2ed1353ce41897b1c86ee286a4f23376ca8c26cabca2396c7cc836e92f37db95"}
May 20 15:32:33.075957 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:33.075354    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-22rt8"
May 20 15:32:33.076625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:33.076604    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zcsn" event={"ID":"e5078f38-1131-44ce-8880-29fe9ccfd946","Type":"ContainerStarted","Data":"ae9546a781bb489fd9b9a3b1518ee4b157318a13f72ed5f98fbb915146849ca2"}
May 20 15:32:33.101026 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:33.100982    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-22rt8" podStartSLOduration=16.924571086 podStartE2EDuration="19.100970934s" podCreationTimestamp="2026-05-20 15:32:14 +0000 UTC" firstStartedPulling="2026-05-20 15:32:30.315152635 +0000 UTC m=+49.055903291" lastFinishedPulling="2026-05-20 15:32:32.491552483 +0000 UTC m=+51.232303139" observedRunningTime="2026-05-20 15:32:33.100934951 +0000 UTC m=+51.841685642" watchObservedRunningTime="2026-05-20 15:32:33.100970934 +0000 UTC m=+51.841721615"
May 20 15:32:41.019642 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.019616    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjrxd"
May 20 15:32:41.214495 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.214442    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6zcsn" podStartSLOduration=25.003476719 podStartE2EDuration="27.214426016s" podCreationTimestamp="2026-05-20 15:32:14 +0000 UTC" firstStartedPulling="2026-05-20 15:32:30.282775374 +0000 UTC m=+49.023526030" lastFinishedPulling="2026-05-20 15:32:32.49372467 +0000 UTC m=+51.234475327" observedRunningTime="2026-05-20 15:32:33.127390652 +0000 UTC m=+51.868141329" watchObservedRunningTime="2026-05-20 15:32:41.214426016 +0000 UTC m=+59.955176688"
May 20 15:32:41.604817 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.604788    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"]
May 20 15:32:41.607878 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.607859    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.608720 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.608697    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"]
May 20 15:32:41.610661 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.610644    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.616821 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.616800    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 20 15:32:41.616917 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.616878    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 20 15:32:41.616984 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.616960    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 20 15:32:41.617066 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617050    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 20 15:32:41.617293 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617276    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 20 15:32:41.617293 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617291    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 20 15:32:41.617902 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617884    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 20 15:32:41.618041 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617949    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 20 15:32:41.618160 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.617968    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ndqps\""
May 20 15:32:41.624444 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.624424    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"]
May 20 15:32:41.626222 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.626204    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"]
May 20 15:32:41.626325 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.626313    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.629931 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.629830    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 20 15:32:41.637388 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.637363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsqj\" (UniqueName: \"kubernetes.io/projected/b8aa8c2e-63ce-4506-adad-208016878c59-kube-api-access-vwsqj\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.637468 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.637404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8aa8c2e-63ce-4506-adad-208016878c59-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.654292 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.654269    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"]
May 20 15:32:41.655030 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.655014    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"]
May 20 15:32:41.737833 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.737797    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsqj\" (UniqueName: \"kubernetes.io/projected/b8aa8c2e-63ce-4506-adad-208016878c59-kube-api-access-vwsqj\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.737833 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.737839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.738027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.737866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-klusterlet-config\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.738027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.737884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.738027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.737904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38eddcf4-1336-45dd-ae62-5821248055c3-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.738027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-tmp\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.738178 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.738178 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.738178 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhcb\" (UniqueName: \"kubernetes.io/projected/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-kube-api-access-bjhcb\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.738178 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738145    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8aa8c2e-63ce-4506-adad-208016878c59-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.738178 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.738166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxstd\" (UniqueName: \"kubernetes.io/projected/38eddcf4-1336-45dd-ae62-5821248055c3-kube-api-access-qxstd\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.740742 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.740723    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 20 15:32:41.751546 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.751523    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8aa8c2e-63ce-4506-adad-208016878c59-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.780447 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.780416    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:32:41.784182 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.784155    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 20 15:32:41.793844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.793823    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 20 15:32:41.803287 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.803265    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsqj\" (UniqueName: \"kubernetes.io/projected/b8aa8c2e-63ce-4506-adad-208016878c59-kube-api-access-vwsqj\") pod \"managed-serviceaccount-addon-agent-777fd5c9cd-xg79f\" (UID: \"b8aa8c2e-63ce-4506-adad-208016878c59\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.838640 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838610    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.838640 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838641    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-klusterlet-config\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.838848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838661    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.838848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838678    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38eddcf4-1336-45dd-ae62-5821248055c3-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.838848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838830    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-tmp\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.838999 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838866    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.838999 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.838999 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838929    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhcb\" (UniqueName: \"kubernetes.io/projected/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-kube-api-access-bjhcb\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.838999 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.838987    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxstd\" (UniqueName: \"kubernetes.io/projected/38eddcf4-1336-45dd-ae62-5821248055c3-kube-api-access-qxstd\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.839213 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.839181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-tmp\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.839409 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.839392    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38eddcf4-1336-45dd-ae62-5821248055c3-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.847967 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.847937    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 20 15:32:41.847967 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.847952    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 20 15:32:41.848144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.847988    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 20 15:32:41.848144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.848106    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 20 15:32:41.848144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.848117    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 20 15:32:41.851459 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.851439    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-ca\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.851622 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.851489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.851705 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.851689    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.851745 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.851737    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38eddcf4-1336-45dd-ae62-5821248055c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.851868 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.851852    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-klusterlet-config\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.864001 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.863942    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6xcj5"]
May 20 15:32:41.866974 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.866959    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.875483 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875448    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 15:32:41.875598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875536    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhcb\" (UniqueName: \"kubernetes.io/projected/b76fb1dc-0366-429a-9c0d-baebdc4f3af1-kube-api-access-bjhcb\") pod \"klusterlet-addon-workmgr-84b9d9444d-ks5pl\" (UID: \"b76fb1dc-0366-429a-9c0d-baebdc4f3af1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.875598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875562    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 15:32:41.875598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875591    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 15:32:41.875743 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875628    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 15:32:41.875923 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.875909    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b28xv\""
May 20 15:32:41.879603 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.879555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxstd\" (UniqueName: \"kubernetes.io/projected/38eddcf4-1336-45dd-ae62-5821248055c3-kube-api-access-qxstd\") pod \"cluster-proxy-proxy-agent-6b6756b5f7-rj8fw\" (UID: \"38eddcf4-1336-45dd-ae62-5821248055c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.896703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.896677    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6xcj5"]
May 20 15:32:41.927744 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.927608    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 20 15:32:41.928564 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.928541    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-664d49788c-l46lh"]
May 20 15:32:41.930471 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.930452    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:41.939458 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.939433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.939542 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.939474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f86fa47-b321-4b0b-940d-5160e189d8e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.939542 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.939523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmc7\" (UniqueName: \"kubernetes.io/projected/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-api-access-qsmc7\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.939628 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.939568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f86fa47-b321-4b0b-940d-5160e189d8e8-data-volume\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.939697 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.939677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f86fa47-b321-4b0b-940d-5160e189d8e8-crio-socket\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:41.942836 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.942818    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"
May 20 15:32:41.947254 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.947234    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:41.947346 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.947241    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ndqps\""
May 20 15:32:41.955923 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.955886    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"
May 20 15:32:41.985344 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:41.984680    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-664d49788c-l46lh"]
May 20 15:32:42.040359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040100    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-image-registry-private-configuration\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-tls\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040231    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f86fa47-b321-4b0b-940d-5160e189d8e8-crio-socket\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.040359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-trusted-ca\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-installation-pull-secrets\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040520    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f86fa47-b321-4b0b-940d-5160e189d8e8-crio-socket\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-bound-sa-token\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040613    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040640    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f86fa47-b321-4b0b-940d-5160e189d8e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040668    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmc7\" (UniqueName: \"kubernetes.io/projected/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-api-access-qsmc7\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfd4294a-3da7-4dec-b02a-d7e53992e714-ca-trust-extracted\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-certificates\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwsf\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-kube-api-access-6vwsf\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.040975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.040792    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f86fa47-b321-4b0b-940d-5160e189d8e8-data-volume\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.041874 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.041067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f86fa47-b321-4b0b-940d-5160e189d8e8-data-volume\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.041874 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.041232    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.044194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.044147    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f86fa47-b321-4b0b-940d-5160e189d8e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.068334 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.068254    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmc7\" (UniqueName: \"kubernetes.io/projected/7f86fa47-b321-4b0b-940d-5160e189d8e8-kube-api-access-qsmc7\") pod \"insights-runtime-extractor-6xcj5\" (UID: \"7f86fa47-b321-4b0b-940d-5160e189d8e8\") " pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.133374 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.133257    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw"]
May 20 15:32:42.137405 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:42.137377    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38eddcf4_1336_45dd_ae62_5821248055c3.slice/crio-096faef4f6d72c67a787850de764adfcd1bcb5c769e7b5064089d11c37227254 WatchSource:0}: Error finding container 096faef4f6d72c67a787850de764adfcd1bcb5c769e7b5064089d11c37227254: Status 404 returned error can't find the container with id 096faef4f6d72c67a787850de764adfcd1bcb5c769e7b5064089d11c37227254
May 20 15:32:42.141742 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.141721    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-trusted-ca\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.141798 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.141753    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-installation-pull-secrets\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.141798 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.141769    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-bound-sa-token\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.141798 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.141792    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfd4294a-3da7-4dec-b02a-d7e53992e714-ca-trust-extracted\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.142313 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.142281    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-certificates\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.142420 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.142371    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwsf\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-kube-api-access-6vwsf\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.142484 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.142419    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-image-registry-private-configuration\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.142484 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.142454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfd4294a-3da7-4dec-b02a-d7e53992e714-ca-trust-extracted\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.142589 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.142497    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-tls\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.143136 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.143107    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-certificates\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.143419 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.143385    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4294a-3da7-4dec-b02a-d7e53992e714-trusted-ca\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.145233 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.145020    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-installation-pull-secrets\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.147153 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.145720    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-registry-tls\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.147153 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.145748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfd4294a-3da7-4dec-b02a-d7e53992e714-image-registry-private-configuration\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.164511 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.164480    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwsf\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-kube-api-access-6vwsf\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.165366 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.165347    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfd4294a-3da7-4dec-b02a-d7e53992e714-bound-sa-token\") pod \"image-registry-664d49788c-l46lh\" (UID: \"bfd4294a-3da7-4dec-b02a-d7e53992e714\") " pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.175411 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.175389    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6xcj5"
May 20 15:32:42.241014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.240986    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:42.322302 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.322267    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6xcj5"]
May 20 15:32:42.326657 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:42.326622    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f86fa47_b321_4b0b_940d_5160e189d8e8.slice/crio-a290ca09219e3de29b56537fec0957f6d409f0bf5b6d59c1c925afc029451444 WatchSource:0}: Error finding container a290ca09219e3de29b56537fec0957f6d409f0bf5b6d59c1c925afc029451444: Status 404 returned error can't find the container with id a290ca09219e3de29b56537fec0957f6d409f0bf5b6d59c1c925afc029451444
May 20 15:32:42.344477 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.344319    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f"]
May 20 15:32:42.349140 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.347681    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"]
May 20 15:32:42.350126 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:42.349753    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8aa8c2e_63ce_4506_adad_208016878c59.slice/crio-639c9b5c856fddace5ccee0737934bd0cdf15c2f540fcbb2716da8aabce186f9 WatchSource:0}: Error finding container 639c9b5c856fddace5ccee0737934bd0cdf15c2f540fcbb2716da8aabce186f9: Status 404 returned error can't find the container with id 639c9b5c856fddace5ccee0737934bd0cdf15c2f540fcbb2716da8aabce186f9
May 20 15:32:42.359637 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:42.359605    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76fb1dc_0366_429a_9c0d_baebdc4f3af1.slice/crio-1273cb82e640eefce42e5a2b1d78abad4aa8e59eb3ef83cf6f75d221c71a721c WatchSource:0}: Error finding container 1273cb82e640eefce42e5a2b1d78abad4aa8e59eb3ef83cf6f75d221c71a721c: Status 404 returned error can't find the container with id 1273cb82e640eefce42e5a2b1d78abad4aa8e59eb3ef83cf6f75d221c71a721c
May 20 15:32:42.396149 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:42.396031    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-664d49788c-l46lh"]
May 20 15:32:42.398475 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:42.398450    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd4294a_3da7_4dec_b02a_d7e53992e714.slice/crio-0ddb11e61d4ef38007a5dd19f9c35351d8dcfa14a1f9a8b4a8ae1053de8dabb5 WatchSource:0}: Error finding container 0ddb11e61d4ef38007a5dd19f9c35351d8dcfa14a1f9a8b4a8ae1053de8dabb5: Status 404 returned error can't find the container with id 0ddb11e61d4ef38007a5dd19f9c35351d8dcfa14a1f9a8b4a8ae1053de8dabb5
May 20 15:32:43.082881 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.081962    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-22rt8"
May 20 15:32:43.109786 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.109028    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d49788c-l46lh" event={"ID":"bfd4294a-3da7-4dec-b02a-d7e53992e714","Type":"ContainerStarted","Data":"1177bcde426cdd4b88ef0c5a079ae4fc54729f0df10ff0ec2412aedc165fd72d"}
May 20 15:32:43.109786 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.109137    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d49788c-l46lh" event={"ID":"bfd4294a-3da7-4dec-b02a-d7e53992e714","Type":"ContainerStarted","Data":"0ddb11e61d4ef38007a5dd19f9c35351d8dcfa14a1f9a8b4a8ae1053de8dabb5"}
May 20 15:32:43.109786 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.109741    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:32:43.111932 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.111881    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6xcj5" event={"ID":"7f86fa47-b321-4b0b-940d-5160e189d8e8","Type":"ContainerStarted","Data":"1cd49e3ed76d41a0b19ed26694630b82f82fa44046382c9b46e1ef5e487272b9"}
May 20 15:32:43.111932 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.111910    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6xcj5" event={"ID":"7f86fa47-b321-4b0b-940d-5160e189d8e8","Type":"ContainerStarted","Data":"a290ca09219e3de29b56537fec0957f6d409f0bf5b6d59c1c925afc029451444"}
May 20 15:32:43.113741 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.113692    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw" event={"ID":"38eddcf4-1336-45dd-ae62-5821248055c3","Type":"ContainerStarted","Data":"096faef4f6d72c67a787850de764adfcd1bcb5c769e7b5064089d11c37227254"}
May 20 15:32:43.117153 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.117101    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f" event={"ID":"b8aa8c2e-63ce-4506-adad-208016878c59","Type":"ContainerStarted","Data":"639c9b5c856fddace5ccee0737934bd0cdf15c2f540fcbb2716da8aabce186f9"}
May 20 15:32:43.120567 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.120534    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl" event={"ID":"b76fb1dc-0366-429a-9c0d-baebdc4f3af1","Type":"ContainerStarted","Data":"1273cb82e640eefce42e5a2b1d78abad4aa8e59eb3ef83cf6f75d221c71a721c"}
May 20 15:32:43.190440 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:43.189866    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-664d49788c-l46lh" podStartSLOduration=2.189848268 podStartE2EDuration="2.189848268s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:43.187589511 +0000 UTC m=+61.928340190" watchObservedRunningTime="2026-05-20 15:32:43.189848268 +0000 UTC m=+61.930598946"
May 20 15:32:44.130388 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:44.130301    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6xcj5" event={"ID":"7f86fa47-b321-4b0b-940d-5160e189d8e8","Type":"ContainerStarted","Data":"16a2da7f6420780b8140213096bd0b687c109fa2757ea92dda6287edae3c7e7c"}
May 20 15:32:46.587184 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.587146    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:46.590035 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.590013    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:46.600421 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.600390    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4f292c-23eb-45d1-be14-7e368f49ff38-metrics-certs\") pod \"network-metrics-daemon-qmkpv\" (UID: \"9e4f292c-23eb-45d1-be14-7e368f49ff38\") " pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:46.677588 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.677369    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm9j9\""
May 20 15:32:46.686820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.686143    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qmkpv"
May 20 15:32:46.688838 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.688361    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:46.690912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.690885    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:46.700907 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.700890    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:46.712461 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.712432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zf8\" (UniqueName: \"kubernetes.io/projected/64cd1e6e-498b-43df-9ebc-cdc2771a6ad2-kube-api-access-56zf8\") pod \"network-check-target-dz424\" (UID: \"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2\") " pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:46.971257 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.971190    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9zz48\""
May 20 15:32:46.979332 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:46.979302    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:49.917026 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:49.916975    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qmkpv"]
May 20 15:32:49.921964 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:49.921828    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4f292c_23eb_45d1_be14_7e368f49ff38.slice/crio-5a2a42d29d97db3993db698ac6d245a50c5d4a00a379caeb193a1f88aab5dd7e WatchSource:0}: Error finding container 5a2a42d29d97db3993db698ac6d245a50c5d4a00a379caeb193a1f88aab5dd7e: Status 404 returned error can't find the container with id 5a2a42d29d97db3993db698ac6d245a50c5d4a00a379caeb193a1f88aab5dd7e
May 20 15:32:49.944280 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:49.944251    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dz424"]
May 20 15:32:49.947095 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:32:49.947063    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cd1e6e_498b_43df_9ebc_cdc2771a6ad2.slice/crio-9f7b8696e3a1e35c54373f550580c20b3bfb6fe2510a86c03e4280186b62180a WatchSource:0}: Error finding container 9f7b8696e3a1e35c54373f550580c20b3bfb6fe2510a86c03e4280186b62180a: Status 404 returned error can't find the container with id 9f7b8696e3a1e35c54373f550580c20b3bfb6fe2510a86c03e4280186b62180a
May 20 15:32:50.147134 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.147097    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dz424" event={"ID":"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2","Type":"ContainerStarted","Data":"9f7b8696e3a1e35c54373f550580c20b3bfb6fe2510a86c03e4280186b62180a"}
May 20 15:32:50.148535 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.148503    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f" event={"ID":"b8aa8c2e-63ce-4506-adad-208016878c59","Type":"ContainerStarted","Data":"a2b5e62754ef995856afb93afaddc0cb9aa04d212e4293ddfa2e6d1a973aa112"}
May 20 15:32:50.149801 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.149779    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl" event={"ID":"b76fb1dc-0366-429a-9c0d-baebdc4f3af1","Type":"ContainerStarted","Data":"ee4ab61118ba7a3e518fbca28c86b2133480327b76a5d84f66d006bf9d649a3b"}
May 20 15:32:50.149959 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.149942    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:50.150839 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.150810    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmkpv" event={"ID":"9e4f292c-23eb-45d1-be14-7e368f49ff38","Type":"ContainerStarted","Data":"5a2a42d29d97db3993db698ac6d245a50c5d4a00a379caeb193a1f88aab5dd7e"}
May 20 15:32:50.155375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.155354    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl"
May 20 15:32:50.156097 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.156063    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6xcj5" event={"ID":"7f86fa47-b321-4b0b-940d-5160e189d8e8","Type":"ContainerStarted","Data":"44d71bac971bcfce9a4449e77b86247edb9c749edd0c4864021196de6c2dcb2d"}
May 20 15:32:50.157218 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.157200    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw" event={"ID":"38eddcf4-1336-45dd-ae62-5821248055c3","Type":"ContainerStarted","Data":"92a694862a6af33c1750b73a40f1715c1cc5ef65b87d4ed35655d5677b6df65f"}
May 20 15:32:50.167777 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.167731    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-777fd5c9cd-xg79f" podStartSLOduration=1.72636802 podStartE2EDuration="9.16772083s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:42.353903002 +0000 UTC m=+61.094653661" lastFinishedPulling="2026-05-20 15:32:49.795255796 +0000 UTC m=+68.536006471" observedRunningTime="2026-05-20 15:32:50.167170059 +0000 UTC m=+68.907920737" watchObservedRunningTime="2026-05-20 15:32:50.16772083 +0000 UTC m=+68.908471507"
May 20 15:32:50.186232 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.186191    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84b9d9444d-ks5pl" podStartSLOduration=1.7783862780000002 podStartE2EDuration="9.186176423s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:42.361540923 +0000 UTC m=+61.102291579" lastFinishedPulling="2026-05-20 15:32:49.769331054 +0000 UTC m=+68.510081724" observedRunningTime="2026-05-20 15:32:50.185736655 +0000 UTC m=+68.926487337" watchObservedRunningTime="2026-05-20 15:32:50.186176423 +0000 UTC m=+68.926927125"
May 20 15:32:50.209295 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.209237    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6xcj5" podStartSLOduration=1.798655643 podStartE2EDuration="9.209220949s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:42.395610024 +0000 UTC m=+61.136360687" lastFinishedPulling="2026-05-20 15:32:49.806175335 +0000 UTC m=+68.546925993" observedRunningTime="2026-05-20 15:32:50.208622018 +0000 UTC m=+68.949372696" watchObservedRunningTime="2026-05-20 15:32:50.209220949 +0000 UTC m=+68.949971627"
May 20 15:32:50.847365 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.845946    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cm7ht"]
May 20 15:32:50.850537 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.848524    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.854637 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.853838    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8vs4\""
May 20 15:32:50.854637 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.854220    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 15:32:50.854637 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.854430    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:32:50.859288 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.856449    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 15:32:50.859288 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.856698    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 15:32:50.859288 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.856890    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:32:50.859288 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.857189    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.925956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-metrics-client-ca\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflbw\" (UniqueName: \"kubernetes.io/projected/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-kube-api-access-zflbw\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926145    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-textfile\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-root\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-wtmp\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:50.926385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:50.926251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-sys\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027417    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027475    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-textfile\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027507    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-root\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027531    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-wtmp\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027561    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-sys\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027592    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027663    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-metrics-client-ca\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027694    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.027725    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zflbw\" (UniqueName: \"kubernetes.io/projected/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-kube-api-access-zflbw\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.028180    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-sys\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028836 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.028451    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-textfile\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028836 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.028518    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-root\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.028836 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.028623    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-wtmp\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.029146 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.029111    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-metrics-client-ca\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.029247 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:51.029221    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 15:32:51.029300 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:32:51.029294    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls podName:79c9f96f-9d52-4ff5-bec1-603d7211aa3f nodeName:}" failed. No retries permitted until 2026-05-20 15:32:51.52927453 +0000 UTC m=+70.270025200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls") pod "node-exporter-cm7ht" (UID: "79c9f96f-9d52-4ff5-bec1-603d7211aa3f") : secret "node-exporter-tls" not found
May 20 15:32:51.030011 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.029984    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.034039 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.031951    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.037276 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.037233    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflbw\" (UniqueName: \"kubernetes.io/projected/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-kube-api-access-zflbw\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.533803 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.533717    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.536162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.536138    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c9f96f-9d52-4ff5-bec1-603d7211aa3f-node-exporter-tls\") pod \"node-exporter-cm7ht\" (UID: \"79c9f96f-9d52-4ff5-bec1-603d7211aa3f\") " pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.768761 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.768343    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cm7ht"
May 20 15:32:51.787624 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:51.787599    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:32:52.166664 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:52.166623    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmkpv" event={"ID":"9e4f292c-23eb-45d1-be14-7e368f49ff38","Type":"ContainerStarted","Data":"c11c3e16b8b717438a80afc24f7e627bd648cff8cd2a17d1675ec86ed6a91e4c"}
May 20 15:32:52.167145 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:52.166671    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qmkpv" event={"ID":"9e4f292c-23eb-45d1-be14-7e368f49ff38","Type":"ContainerStarted","Data":"41d166d00d6b880d80a8294779cd703a813a74c8e04667b875a72929b238c2f2"}
May 20 15:32:52.167889 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:52.167839    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cm7ht" event={"ID":"79c9f96f-9d52-4ff5-bec1-603d7211aa3f","Type":"ContainerStarted","Data":"e0b3a1fc92aa8a7e90bcb84b1cf421585ea7d0def7e74687091c03e871d496a4"}
May 20 15:32:52.199329 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:52.199274    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qmkpv" podStartSLOduration=68.867157399 podStartE2EDuration="1m10.199257011s" podCreationTimestamp="2026-05-20 15:31:42 +0000 UTC" firstStartedPulling="2026-05-20 15:32:49.924485127 +0000 UTC m=+68.665235783" lastFinishedPulling="2026-05-20 15:32:51.256584723 +0000 UTC m=+69.997335395" observedRunningTime="2026-05-20 15:32:52.197690184 +0000 UTC m=+70.938440862" watchObservedRunningTime="2026-05-20 15:32:52.199257011 +0000 UTC m=+70.940007693"
May 20 15:32:53.172758 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:53.172721    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dz424" event={"ID":"64cd1e6e-498b-43df-9ebc-cdc2771a6ad2","Type":"ContainerStarted","Data":"3441c7fb12bae796b4cd0e1077797ab75bced970dd5c10420cf1b17434112384"}
May 20 15:32:53.191243 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:53.191189    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dz424" podStartSLOduration=69.128852093 podStartE2EDuration="1m12.191160756s" podCreationTimestamp="2026-05-20 15:31:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:49.949339502 +0000 UTC m=+68.690090173" lastFinishedPulling="2026-05-20 15:32:53.01164818 +0000 UTC m=+71.752398836" observedRunningTime="2026-05-20 15:32:53.189094983 +0000 UTC m=+71.929845656" watchObservedRunningTime="2026-05-20 15:32:53.191160756 +0000 UTC m=+71.931911435"
May 20 15:32:54.176886 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.176792    2574 generic.go:358] "Generic (PLEG): container finished" podID="79c9f96f-9d52-4ff5-bec1-603d7211aa3f" containerID="38b15994ab1e0c17f5398c951d5e91d61079936d95a504093efa810af2d91dd2" exitCode=0
May 20 15:32:54.177335 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.176884    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cm7ht" event={"ID":"79c9f96f-9d52-4ff5-bec1-603d7211aa3f","Type":"ContainerDied","Data":"38b15994ab1e0c17f5398c951d5e91d61079936d95a504093efa810af2d91dd2"}
May 20 15:32:54.182350 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.181851    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw" event={"ID":"38eddcf4-1336-45dd-ae62-5821248055c3","Type":"ContainerStarted","Data":"c6a66d01f0b1d1e0249d382f71f7ee1999c058a185de4b03a0eb51f298ae90b9"}
May 20 15:32:54.182491 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.182353    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:32:54.182491 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.182448    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw" event={"ID":"38eddcf4-1336-45dd-ae62-5821248055c3","Type":"ContainerStarted","Data":"6998fadf428d1b817f8798b5641b962c8a83814a42850ee30c5ec501a5ac3a91"}
May 20 15:32:54.222233 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:54.222189    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b6756b5f7-rj8fw" podStartSLOduration=1.515082605 podStartE2EDuration="13.222171934s" podCreationTimestamp="2026-05-20 15:32:41 +0000 UTC" firstStartedPulling="2026-05-20 15:32:42.139132695 +0000 UTC m=+60.879883350" lastFinishedPulling="2026-05-20 15:32:53.846222024 +0000 UTC m=+72.586972679" observedRunningTime="2026-05-20 15:32:54.221169065 +0000 UTC m=+72.961919744" watchObservedRunningTime="2026-05-20 15:32:54.222171934 +0000 UTC m=+72.962922613"
May 20 15:32:55.186159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:55.186119    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cm7ht" event={"ID":"79c9f96f-9d52-4ff5-bec1-603d7211aa3f","Type":"ContainerStarted","Data":"24063e2b6b19dccf64deae2090fba4686c4ee450ddfca17822a79ff80eb3a7c2"}
May 20 15:32:55.186159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:55.186162    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cm7ht" event={"ID":"79c9f96f-9d52-4ff5-bec1-603d7211aa3f","Type":"ContainerStarted","Data":"89b80932bfc041c7df30bfb2d560752b519bbe9fb9fe210033f1772186d31b0c"}
May 20 15:32:55.206729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:32:55.206680    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cm7ht" podStartSLOduration=3.144428139 podStartE2EDuration="5.206666234s" podCreationTimestamp="2026-05-20 15:32:50 +0000 UTC" firstStartedPulling="2026-05-20 15:32:51.783934558 +0000 UTC m=+70.524685237" lastFinishedPulling="2026-05-20 15:32:53.84617267 +0000 UTC m=+72.586923332" observedRunningTime="2026-05-20 15:32:55.204759911 +0000 UTC m=+73.945510586" watchObservedRunningTime="2026-05-20 15:32:55.206666234 +0000 UTC m=+73.947416911"
May 20 15:33:05.137729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:05.137624    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-664d49788c-l46lh"
May 20 15:33:06.800273 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:06.800203    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" podUID="d852f365-9ffe-4dd9-a72b-dfe53d974547" containerName="registry" containerID="cri-o://3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57" gracePeriod=30
May 20 15:33:07.035665 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.035642    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:33:07.162815 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.162782    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.162978 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.162824    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.162978 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.162859    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163054 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163029    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163156 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163135    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163334 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163307    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:07.163452 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163355    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163452 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163377    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg9cx\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163452 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163431    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca\") pod \"d852f365-9ffe-4dd9-a72b-dfe53d974547\" (UID: \"d852f365-9ffe-4dd9-a72b-dfe53d974547\") "
May 20 15:33:07.163668 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.163648    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-certificates\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.164133 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.164106    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:07.165701 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.165659    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:07.165796 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.165709    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:07.165796 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.165759    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:07.165872 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.165800    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:07.166375 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.166348    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx" (OuterVolumeSpecName: "kube-api-access-cg9cx") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "kube-api-access-cg9cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:07.172887 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.172862    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d852f365-9ffe-4dd9-a72b-dfe53d974547" (UID: "d852f365-9ffe-4dd9-a72b-dfe53d974547"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:33:07.218966 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.218938    2574 generic.go:358] "Generic (PLEG): container finished" podID="d852f365-9ffe-4dd9-a72b-dfe53d974547" containerID="3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57" exitCode=0
May 20 15:33:07.219101 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.219011    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" event={"ID":"d852f365-9ffe-4dd9-a72b-dfe53d974547","Type":"ContainerDied","Data":"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"}
May 20 15:33:07.219101 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.219015    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm"
May 20 15:33:07.219101 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.219038    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85768dcc7d-f5rdm" event={"ID":"d852f365-9ffe-4dd9-a72b-dfe53d974547","Type":"ContainerDied","Data":"fad661cc219f78296ce2163b3b45da8353cf62fe9e1f9b06dcc8dab3b473a3f0"}
May 20 15:33:07.219101 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.219057    2574 scope.go:117] "RemoveContainer" containerID="3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"
May 20 15:33:07.226727 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.226707    2574 scope.go:117] "RemoveContainer" containerID="3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"
May 20 15:33:07.226979 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:33:07.226952    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57\": container with ID starting with 3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57 not found: ID does not exist" containerID="3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"
May 20 15:33:07.227061 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.226981    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57"} err="failed to get container status \"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57\": rpc error: code = NotFound desc = could not find container \"3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57\": container with ID starting with 3fa7b9930145e62ac3eefc33ce90150a04d9bd52241fa7277f9c12cb6a218f57 not found: ID does not exist"
May 20 15:33:07.239521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.239497    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:33:07.242203 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.242184    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85768dcc7d-f5rdm"]
May 20 15:33:07.264714 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264693    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-bound-sa-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264714 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264712    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-installation-pull-secrets\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264724    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d852f365-9ffe-4dd9-a72b-dfe53d974547-ca-trust-extracted\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264734    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg9cx\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-kube-api-access-cg9cx\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264743    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d852f365-9ffe-4dd9-a72b-dfe53d974547-trusted-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264751    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d852f365-9ffe-4dd9-a72b-dfe53d974547-registry-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.264831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.264760    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d852f365-9ffe-4dd9-a72b-dfe53d974547-image-registry-private-configuration\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:33:07.861432 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:07.861399    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d852f365-9ffe-4dd9-a72b-dfe53d974547" path="/var/lib/kubelet/pods/d852f365-9ffe-4dd9-a72b-dfe53d974547/volumes"
May 20 15:33:25.188991 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:33:25.188957    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dz424"
May 20 15:36:41.774076 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:36:41.774049    2574 kubelet.go:1628] "Image garbage collection succeeded"
May 20 15:40:59.038196 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.038166    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-8qkdq"]
May 20 15:40:59.038575 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.038385    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d852f365-9ffe-4dd9-a72b-dfe53d974547" containerName="registry"
May 20 15:40:59.038575 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.038395    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d852f365-9ffe-4dd9-a72b-dfe53d974547" containerName="registry"
May 20 15:40:59.038575 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.038445    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="d852f365-9ffe-4dd9-a72b-dfe53d974547" containerName="registry"
May 20 15:40:59.039942 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.039927    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.042328 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.042306    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 20 15:40:59.043266 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.043251    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mm7q4\""
May 20 15:40:59.043319 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.043256    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 20 15:40:59.049529 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.049509    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-8qkdq"]
May 20 15:40:59.224546 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.224520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgbm\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-kube-api-access-8zgbm\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.224707 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.224554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.325870 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.325799    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgbm\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-kube-api-access-8zgbm\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.325870 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.325831    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.335005 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.334982    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.335005 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.335005    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgbm\" (UniqueName: \"kubernetes.io/projected/673732ea-baa6-4786-ae57-ba576f19443c-kube-api-access-8zgbm\") pod \"cert-manager-cainjector-68b757865b-8qkdq\" (UID: \"673732ea-baa6-4786-ae57-ba576f19443c\") " pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.349031 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.349012    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq"
May 20 15:40:59.473304 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.473277    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-8qkdq"]
May 20 15:40:59.476382 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:40:59.476356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod673732ea_baa6_4786_ae57_ba576f19443c.slice/crio-2cbc4ce12fc26962fea538e123671e67d2a2b033e962e09f8a334f0ec14cd3b0 WatchSource:0}: Error finding container 2cbc4ce12fc26962fea538e123671e67d2a2b033e962e09f8a334f0ec14cd3b0: Status 404 returned error can't find the container with id 2cbc4ce12fc26962fea538e123671e67d2a2b033e962e09f8a334f0ec14cd3b0
May 20 15:40:59.478027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:40:59.478009    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:41:00.382105 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:00.382057    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq" event={"ID":"673732ea-baa6-4786-ae57-ba576f19443c","Type":"ContainerStarted","Data":"2cbc4ce12fc26962fea538e123671e67d2a2b033e962e09f8a334f0ec14cd3b0"}
May 20 15:41:04.392429 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:04.392393    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq" event={"ID":"673732ea-baa6-4786-ae57-ba576f19443c","Type":"ContainerStarted","Data":"e41eb1f563704237b1f725328a56d5c26eede97cf30731cad3ae45eebd9c0bf6"}
May 20 15:41:04.409499 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:04.409439    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-8qkdq" podStartSLOduration=1.007097364 podStartE2EDuration="5.409422839s" podCreationTimestamp="2026-05-20 15:40:59 +0000 UTC" firstStartedPulling="2026-05-20 15:40:59.478180164 +0000 UTC m=+558.218930820" lastFinishedPulling="2026-05-20 15:41:03.880505634 +0000 UTC m=+562.621256295" observedRunningTime="2026-05-20 15:41:04.408540661 +0000 UTC m=+563.149291334" watchObservedRunningTime="2026-05-20 15:41:04.409422839 +0000 UTC m=+563.150173518"
May 20 15:41:15.910364 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.910332    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-8lw9v"]
May 20 15:41:15.912302 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.912287    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:15.914822 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.914805    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-twcn8\""
May 20 15:41:15.923305 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.923286    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-8lw9v"]
May 20 15:41:15.937618 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.937594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cltq\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-kube-api-access-9cltq\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:15.937713 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:15.937628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-bound-sa-token\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.038404 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.038380    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9cltq\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-kube-api-access-9cltq\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.038489 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.038419    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-bound-sa-token\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.048019 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.047994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-bound-sa-token\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.048126 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.048113    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cltq\" (UniqueName: \"kubernetes.io/projected/e56518ad-30e6-491f-9081-900edde0056c-kube-api-access-9cltq\") pod \"cert-manager-79c8d999ff-8lw9v\" (UID: \"e56518ad-30e6-491f-9081-900edde0056c\") " pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.220426 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.220367    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-8lw9v"
May 20 15:41:16.340897 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.340864    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-8lw9v"]
May 20 15:41:16.343753 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:41:16.343728    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode56518ad_30e6_491f_9081_900edde0056c.slice/crio-f133ce2f2e9060366be5342d979b2ad3d4abf6dd9efc6b731931d6a4c31c4ead WatchSource:0}: Error finding container f133ce2f2e9060366be5342d979b2ad3d4abf6dd9efc6b731931d6a4c31c4ead: Status 404 returned error can't find the container with id f133ce2f2e9060366be5342d979b2ad3d4abf6dd9efc6b731931d6a4c31c4ead
May 20 15:41:16.425041 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.425007    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-8lw9v" event={"ID":"e56518ad-30e6-491f-9081-900edde0056c","Type":"ContainerStarted","Data":"4641d326763d86b6b463f98e49e7e16fce5353d6d1d48ca0abc836a0655b7067"}
May 20 15:41:16.425179 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.425050    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-8lw9v" event={"ID":"e56518ad-30e6-491f-9081-900edde0056c","Type":"ContainerStarted","Data":"f133ce2f2e9060366be5342d979b2ad3d4abf6dd9efc6b731931d6a4c31c4ead"}
May 20 15:41:16.444931 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:16.444890    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-8lw9v" podStartSLOduration=1.4448764619999999 podStartE2EDuration="1.444876462s" podCreationTimestamp="2026-05-20 15:41:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:41:16.443522787 +0000 UTC m=+575.184273465" watchObservedRunningTime="2026-05-20 15:41:16.444876462 +0000 UTC m=+575.185627139"
May 20 15:41:40.210754 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.210717    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"]
May 20 15:41:40.212816 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.212798    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.216981 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.216959    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 20 15:41:40.217120 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.217001    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 20 15:41:40.217523 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.217508    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 20 15:41:40.218041 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.218023    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 20 15:41:40.218146 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.218050    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xd2hj\""
May 20 15:41:40.218849 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.218830    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 20 15:41:40.227930 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.227909    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"]
May 20 15:41:40.300126 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.300080    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r42z\" (UniqueName: \"kubernetes.io/projected/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-kube-api-access-5r42z\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.300246 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.300143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-manager-config\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.300246 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.300176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.300246 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.300228    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-metrics-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.401134 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.401109    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-metrics-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.401216 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.401144    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5r42z\" (UniqueName: \"kubernetes.io/projected/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-kube-api-access-5r42z\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.401216 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.401169    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-manager-config\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.401216 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.401205    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.401773 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.401755    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-manager-config\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.403592 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.403573    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.403592 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.403586    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-metrics-cert\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.410882 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.410864    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r42z\" (UniqueName: \"kubernetes.io/projected/d1a71f86-c7ec-4feb-b398-cd925eb5bc6d-kube-api-access-5r42z\") pod \"lws-controller-manager-cf55c5997-2xlk9\" (UID: \"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d\") " pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.522126 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.522044    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:40.663406 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:40.663377    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"]
May 20 15:41:40.672537 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:41:40.672510    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a71f86_c7ec_4feb_b398_cd925eb5bc6d.slice/crio-88679a182497b4a213554eca6fc89596edaafba4e7e302c1ca114a8097002120 WatchSource:0}: Error finding container 88679a182497b4a213554eca6fc89596edaafba4e7e302c1ca114a8097002120: Status 404 returned error can't find the container with id 88679a182497b4a213554eca6fc89596edaafba4e7e302c1ca114a8097002120
May 20 15:41:41.489352 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:41.489319    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9" event={"ID":"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d","Type":"ContainerStarted","Data":"88679a182497b4a213554eca6fc89596edaafba4e7e302c1ca114a8097002120"}
May 20 15:41:45.502362 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:45.502281    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9" event={"ID":"d1a71f86-c7ec-4feb-b398-cd925eb5bc6d","Type":"ContainerStarted","Data":"6982710a83ff769e7735d7f049a11f009618b8d3997e1ac2dac31b7cf358414a"}
May 20 15:41:45.502696 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:45.502381    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:41:45.520266 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:45.520225    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9" podStartSLOduration=1.097128573 podStartE2EDuration="5.520213395s" podCreationTimestamp="2026-05-20 15:41:40 +0000 UTC" firstStartedPulling="2026-05-20 15:41:40.674502092 +0000 UTC m=+599.415252748" lastFinishedPulling="2026-05-20 15:41:45.097586904 +0000 UTC m=+603.838337570" observedRunningTime="2026-05-20 15:41:45.517877225 +0000 UTC m=+604.258627903" watchObservedRunningTime="2026-05-20 15:41:45.520213395 +0000 UTC m=+604.260964072"
May 20 15:41:56.507593 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:41:56.507564    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cf55c5997-2xlk9"
May 20 15:42:49.631528 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.631494    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"]
May 20 15:42:49.633526 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.633510    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.636436 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.636405    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 20 15:42:49.636558 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.636437    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 20 15:42:49.636558 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.636405    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnrgp\""
May 20 15:42:49.637280 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.637265    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 20 15:42:49.637367 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.637306    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 20 15:42:49.643703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.643686    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"]
May 20 15:42:49.765282 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.765255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.765441 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.765293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn4d\" (UniqueName: \"kubernetes.io/projected/7b261504-b677-4bd0-b497-c29253d3ff3e-kube-api-access-lcn4d\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.765441 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.765365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b261504-b677-4bd0-b497-c29253d3ff3e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.865923 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.865889    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b261504-b677-4bd0-b497-c29253d3ff3e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.866072 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.865937    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.866072 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.865969    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn4d\" (UniqueName: \"kubernetes.io/projected/7b261504-b677-4bd0-b497-c29253d3ff3e-kube-api-access-lcn4d\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.866223 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:42:49.866077    2574 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 20 15:42:49.866223 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:42:49.866155    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert podName:7b261504-b677-4bd0-b497-c29253d3ff3e nodeName:}" failed. No retries permitted until 2026-05-20 15:42:50.366140477 +0000 UTC m=+669.106891134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-kkwsz" (UID: "7b261504-b677-4bd0-b497-c29253d3ff3e") : secret "plugin-serving-cert" not found
May 20 15:42:49.866443 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.866425    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b261504-b677-4bd0-b497-c29253d3ff3e-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:49.886113 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:49.886040    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn4d\" (UniqueName: \"kubernetes.io/projected/7b261504-b677-4bd0-b497-c29253d3ff3e-kube-api-access-lcn4d\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:50.369287 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:50.369263    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:50.371537 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:50.371509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b261504-b677-4bd0-b497-c29253d3ff3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-kkwsz\" (UID: \"7b261504-b677-4bd0-b497-c29253d3ff3e\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:50.543352 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:50.543328    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"
May 20 15:42:50.662869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:50.662845    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz"]
May 20 15:42:50.665179 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:42:50.665144    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b261504_b677_4bd0_b497_c29253d3ff3e.slice/crio-e4a1488972de1963d218bbc755fed6d0836edc3cd741ab24fc1ef539fe7de7de WatchSource:0}: Error finding container e4a1488972de1963d218bbc755fed6d0836edc3cd741ab24fc1ef539fe7de7de: Status 404 returned error can't find the container with id e4a1488972de1963d218bbc755fed6d0836edc3cd741ab24fc1ef539fe7de7de
May 20 15:42:50.670835 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:50.670799    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz" event={"ID":"7b261504-b677-4bd0-b497-c29253d3ff3e","Type":"ContainerStarted","Data":"e4a1488972de1963d218bbc755fed6d0836edc3cd741ab24fc1ef539fe7de7de"}
May 20 15:42:55.684909 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:55.684866    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz" event={"ID":"7b261504-b677-4bd0-b497-c29253d3ff3e","Type":"ContainerStarted","Data":"8558d254f10938af13f7b11de85d0ad228592fefc1073a3679bfeae1f92b082e"}
May 20 15:42:55.700832 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:42:55.700783    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-kkwsz" podStartSLOduration=2.447109502 podStartE2EDuration="6.700770678s" podCreationTimestamp="2026-05-20 15:42:49 +0000 UTC" firstStartedPulling="2026-05-20 15:42:50.666316882 +0000 UTC m=+669.407067537" lastFinishedPulling="2026-05-20 15:42:54.919978056 +0000 UTC m=+673.660728713" observedRunningTime="2026-05-20 15:42:55.700683912 +0000 UTC m=+674.441434590" watchObservedRunningTime="2026-05-20 15:42:55.700770678 +0000 UTC m=+674.441521347"
May 20 15:43:29.580762 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.580668    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:29.583823 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.583802    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.587554 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.587529    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 20 15:43:29.613706 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.613680    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:29.630075 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.630045    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:29.637643 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.637621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.637778 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.637756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6nzt\" (UniqueName: \"kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.738963 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.738939    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r6nzt\" (UniqueName: \"kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.739074 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.738978    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.739507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.739488    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.747668 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.747644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6nzt\" (UniqueName: \"kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt\") pod \"limitador-limitador-64c8f475fb-s7wxp\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:29.894470 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:29.894404    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:30.034539 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:43:30.034509    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657913d5_5dbd_476b_b442_3481894847e0.slice/crio-611f2885ab9358015d8c8b779a21a46d89bbd57d2c6942fbf4c1a665288cd128 WatchSource:0}: Error finding container 611f2885ab9358015d8c8b779a21a46d89bbd57d2c6942fbf4c1a665288cd128: Status 404 returned error can't find the container with id 611f2885ab9358015d8c8b779a21a46d89bbd57d2c6942fbf4c1a665288cd128
May 20 15:43:30.034975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:30.034898    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:30.777242 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:30.777205    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" event={"ID":"657913d5-5dbd-476b-b442-3481894847e0","Type":"ContainerStarted","Data":"611f2885ab9358015d8c8b779a21a46d89bbd57d2c6942fbf4c1a665288cd128"}
May 20 15:43:32.784336 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:32.784311    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" event={"ID":"657913d5-5dbd-476b-b442-3481894847e0","Type":"ContainerStarted","Data":"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"}
May 20 15:43:32.784606 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:32.784456    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:32.824755 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:32.824712    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" podStartSLOduration=1.163591089 podStartE2EDuration="3.824699184s" podCreationTimestamp="2026-05-20 15:43:29 +0000 UTC" firstStartedPulling="2026-05-20 15:43:30.036223662 +0000 UTC m=+708.776974318" lastFinishedPulling="2026-05-20 15:43:32.697331753 +0000 UTC m=+711.438082413" observedRunningTime="2026-05-20 15:43:32.821197768 +0000 UTC m=+711.561948446" watchObservedRunningTime="2026-05-20 15:43:32.824699184 +0000 UTC m=+711.565449863"
May 20 15:43:43.788811 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:43.788779    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:46.952939 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:46.952909    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:46.953412 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:46.953124    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" podUID="657913d5-5dbd-476b-b442-3481894847e0" containerName="limitador" containerID="cri-o://50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588" gracePeriod=30
May 20 15:43:47.493922 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.493902    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:47.570802 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.570708    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file\") pod \"657913d5-5dbd-476b-b442-3481894847e0\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") "
May 20 15:43:47.570802 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.570749    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6nzt\" (UniqueName: \"kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt\") pod \"657913d5-5dbd-476b-b442-3481894847e0\" (UID: \"657913d5-5dbd-476b-b442-3481894847e0\") "
May 20 15:43:47.571157 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.571132    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file" (OuterVolumeSpecName: "config-file") pod "657913d5-5dbd-476b-b442-3481894847e0" (UID: "657913d5-5dbd-476b-b442-3481894847e0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:43:47.572903 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.572876    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt" (OuterVolumeSpecName: "kube-api-access-r6nzt") pod "657913d5-5dbd-476b-b442-3481894847e0" (UID: "657913d5-5dbd-476b-b442-3481894847e0"). InnerVolumeSpecName "kube-api-access-r6nzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:43:47.671333 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.671302    2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/657913d5-5dbd-476b-b442-3481894847e0-config-file\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:43:47.671333 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.671328    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6nzt\" (UniqueName: \"kubernetes.io/projected/657913d5-5dbd-476b-b442-3481894847e0-kube-api-access-r6nzt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:43:47.828715 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.828637    2574 generic.go:358] "Generic (PLEG): container finished" podID="657913d5-5dbd-476b-b442-3481894847e0" containerID="50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588" exitCode=0
May 20 15:43:47.828715 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.828695    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"
May 20 15:43:47.828920 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.828725    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" event={"ID":"657913d5-5dbd-476b-b442-3481894847e0","Type":"ContainerDied","Data":"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"}
May 20 15:43:47.828920 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.828768    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-s7wxp" event={"ID":"657913d5-5dbd-476b-b442-3481894847e0","Type":"ContainerDied","Data":"611f2885ab9358015d8c8b779a21a46d89bbd57d2c6942fbf4c1a665288cd128"}
May 20 15:43:47.828920 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.828791    2574 scope.go:117] "RemoveContainer" containerID="50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"
May 20 15:43:47.836646 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.836630    2574 scope.go:117] "RemoveContainer" containerID="50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"
May 20 15:43:47.836881 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:43:47.836864    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588\": container with ID starting with 50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588 not found: ID does not exist" containerID="50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"
May 20 15:43:47.836933 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.836888    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588"} err="failed to get container status \"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588\": rpc error: code = NotFound desc = could not find container \"50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588\": container with ID starting with 50a5be402876478beb725d6ff4f2759645cf9ab42e443dd7791d10c680bce588 not found: ID does not exist"
May 20 15:43:47.852604 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.852585    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:47.861881 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:47.861860    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-s7wxp"]
May 20 15:43:49.861271 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:43:49.861237    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657913d5-5dbd-476b-b442-3481894847e0" path="/var/lib/kubelet/pods/657913d5-5dbd-476b-b442-3481894847e0/volumes"
May 20 15:44:38.086181 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.086140    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-279fk"]
May 20 15:44:38.086956 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.086506    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657913d5-5dbd-476b-b442-3481894847e0" containerName="limitador"
May 20 15:44:38.086956 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.086519    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="657913d5-5dbd-476b-b442-3481894847e0" containerName="limitador"
May 20 15:44:38.086956 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.086584    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="657913d5-5dbd-476b-b442-3481894847e0" containerName="limitador"
May 20 15:44:38.089325 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.089306    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.091842 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.091825    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 15:44:38.092134 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.092117    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 20 15:44:38.092267 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.092249    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 15:44:38.092849 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.092821    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-z92df\""
May 20 15:44:38.098722 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.098699    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-279fk"]
May 20 15:44:38.200452 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.200428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7flxp\" (UniqueName: \"kubernetes.io/projected/83499bdf-28f6-4ed9-92da-484c629caa50-kube-api-access-7flxp\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.200566 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.200463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.301198 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.301174    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7flxp\" (UniqueName: \"kubernetes.io/projected/83499bdf-28f6-4ed9-92da-484c629caa50-kube-api-access-7flxp\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.301308 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.301211    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.301352 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:44:38.301316    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 20 15:44:38.301390 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:44:38.301383    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert podName:83499bdf-28f6-4ed9-92da-484c629caa50 nodeName:}" failed. No retries permitted until 2026-05-20 15:44:38.801363691 +0000 UTC m=+777.542114353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert") pod "odh-model-controller-696fc77849-279fk" (UID: "83499bdf-28f6-4ed9-92da-484c629caa50") : secret "odh-model-controller-webhook-cert" not found
May 20 15:44:38.312365 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.312342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7flxp\" (UniqueName: \"kubernetes.io/projected/83499bdf-28f6-4ed9-92da-484c629caa50-kube-api-access-7flxp\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.804789 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.804758    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:38.807044 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:38.807018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83499bdf-28f6-4ed9-92da-484c629caa50-cert\") pod \"odh-model-controller-696fc77849-279fk\" (UID: \"83499bdf-28f6-4ed9-92da-484c629caa50\") " pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:39.001381 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:39.001353    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:39.122439 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:39.122411    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-279fk"]
May 20 15:44:39.125639 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:44:39.125612    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83499bdf_28f6_4ed9_92da_484c629caa50.slice/crio-c31c473224bab4184e371b56e52e5c9b08eb5ad4e6b03acb67db5120277c4011 WatchSource:0}: Error finding container c31c473224bab4184e371b56e52e5c9b08eb5ad4e6b03acb67db5120277c4011: Status 404 returned error can't find the container with id c31c473224bab4184e371b56e52e5c9b08eb5ad4e6b03acb67db5120277c4011
May 20 15:44:39.968109 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:39.968051    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-279fk" event={"ID":"83499bdf-28f6-4ed9-92da-484c629caa50","Type":"ContainerStarted","Data":"c31c473224bab4184e371b56e52e5c9b08eb5ad4e6b03acb67db5120277c4011"}
May 20 15:44:41.974113 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:41.974017    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-279fk" event={"ID":"83499bdf-28f6-4ed9-92da-484c629caa50","Type":"ContainerStarted","Data":"93a664248ca6079d5907cc8cea5b6febaa902ea5dad3b1a0fafba02567d9d0a4"}
May 20 15:44:41.974113 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:41.974096    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:41.993298 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:41.993259    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-279fk" podStartSLOduration=1.71684507 podStartE2EDuration="3.99324818s" podCreationTimestamp="2026-05-20 15:44:38 +0000 UTC" firstStartedPulling="2026-05-20 15:44:39.127237023 +0000 UTC m=+777.867987679" lastFinishedPulling="2026-05-20 15:44:41.403640132 +0000 UTC m=+780.144390789" observedRunningTime="2026-05-20 15:44:41.991487096 +0000 UTC m=+780.732237773" watchObservedRunningTime="2026-05-20 15:44:41.99324818 +0000 UTC m=+780.733998889"
May 20 15:44:52.979411 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:52.979383    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-279fk"
May 20 15:44:53.874962 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:53.874934    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jpvfk"]
May 20 15:44:53.882687 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:53.882670    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jpvfk"
May 20 15:44:53.884486 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:53.884456    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jpvfk"]
May 20 15:44:53.885354 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:53.885337    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7wj7m\""
May 20 15:44:53.885431 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:53.885337    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 15:44:54.011014 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:54.010988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ht7j\" (UniqueName: \"kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j\") pod \"s3-init-jpvfk\" (UID: \"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9\") " pod="kserve/s3-init-jpvfk"
May 20 15:44:54.111852 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:54.111831    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8ht7j\" (UniqueName: \"kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j\") pod \"s3-init-jpvfk\" (UID: \"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9\") " pod="kserve/s3-init-jpvfk"
May 20 15:44:54.121017 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:54.120992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ht7j\" (UniqueName: \"kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j\") pod \"s3-init-jpvfk\" (UID: \"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9\") " pod="kserve/s3-init-jpvfk"
May 20 15:44:54.192764 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:54.192717    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jpvfk"
May 20 15:44:54.311250 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:54.311215    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jpvfk"]
May 20 15:44:54.314462 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:44:54.314436    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5430d97e_96a6_4d1c_a1ea_d2bce943a5f9.slice/crio-2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed WatchSource:0}: Error finding container 2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed: Status 404 returned error can't find the container with id 2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed
May 20 15:44:55.010046 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:55.010009    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jpvfk" event={"ID":"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9","Type":"ContainerStarted","Data":"2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed"}
May 20 15:44:59.022847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:59.022774    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jpvfk" event={"ID":"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9","Type":"ContainerStarted","Data":"8d54e85ce913672112c9ba606627f7a5bf073ffce9527b8eac177f487aad8e49"}
May 20 15:44:59.040593 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:44:59.040539    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-jpvfk" podStartSLOduration=1.683507922 podStartE2EDuration="6.040523942s" podCreationTimestamp="2026-05-20 15:44:53 +0000 UTC" firstStartedPulling="2026-05-20 15:44:54.315953247 +0000 UTC m=+793.056703904" lastFinishedPulling="2026-05-20 15:44:58.672969268 +0000 UTC m=+797.413719924" observedRunningTime="2026-05-20 15:44:59.039836042 +0000 UTC m=+797.780586720" watchObservedRunningTime="2026-05-20 15:44:59.040523942 +0000 UTC m=+797.781274621"
May 20 15:45:02.032642 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:02.032611    2574 generic.go:358] "Generic (PLEG): container finished" podID="5430d97e-96a6-4d1c-a1ea-d2bce943a5f9" containerID="8d54e85ce913672112c9ba606627f7a5bf073ffce9527b8eac177f487aad8e49" exitCode=0
May 20 15:45:02.033003 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:02.032663    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jpvfk" event={"ID":"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9","Type":"ContainerDied","Data":"8d54e85ce913672112c9ba606627f7a5bf073ffce9527b8eac177f487aad8e49"}
May 20 15:45:03.160399 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:03.160379    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jpvfk"
May 20 15:45:03.181896 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:03.181873    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ht7j\" (UniqueName: \"kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j\") pod \"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9\" (UID: \"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9\") "
May 20 15:45:03.183868 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:03.183843    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j" (OuterVolumeSpecName: "kube-api-access-8ht7j") pod "5430d97e-96a6-4d1c-a1ea-d2bce943a5f9" (UID: "5430d97e-96a6-4d1c-a1ea-d2bce943a5f9"). InnerVolumeSpecName "kube-api-access-8ht7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:45:03.282580 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:03.282552    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ht7j\" (UniqueName: \"kubernetes.io/projected/5430d97e-96a6-4d1c-a1ea-d2bce943a5f9-kube-api-access-8ht7j\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:45:04.038579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:04.038535    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jpvfk" event={"ID":"5430d97e-96a6-4d1c-a1ea-d2bce943a5f9","Type":"ContainerDied","Data":"2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed"}
May 20 15:45:04.038579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:04.038560    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jpvfk"
May 20 15:45:04.038579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:04.038570    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bb81110b8b4765a48d42cf126e2b33a432bd71c8fdc74c4f9608889cc901bed"
May 20 15:45:16.577376 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.577340    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"]
May 20 15:45:16.577862 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.577720    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5430d97e-96a6-4d1c-a1ea-d2bce943a5f9" containerName="s3-init"
May 20 15:45:16.577862 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.577735    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5430d97e-96a6-4d1c-a1ea-d2bce943a5f9" containerName="s3-init"
May 20 15:45:16.577862 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.577786    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="5430d97e-96a6-4d1c-a1ea-d2bce943a5f9" containerName="s3-init"
May 20 15:45:16.579769 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.579748    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.582272 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.582250    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:45:16.582404 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.582289    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
May 20 15:45:16.582657 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.582633    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:45:16.582760 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.582717    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-php58\""
May 20 15:45:16.594945 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.594926    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"]
May 20 15:45:16.673345 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673319    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673491 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177565c0-e6b6-4421-b395-986c4fc84d80-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673491 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dp8x\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-kube-api-access-2dp8x\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673491 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673659 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673659 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673659 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177565c0-e6b6-4421-b395-986c4fc84d80-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673768 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.673768 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.673694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.774839 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774813    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177565c0-e6b6-4421-b395-986c4fc84d80-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.774961 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774867    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.774961 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774890    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.774961 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774927    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.774961 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774954    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177565c0-e6b6-4421-b395-986c4fc84d80-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775200 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.774985    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2dp8x\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-kube-api-access-2dp8x\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775200 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775020    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775200 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775046    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775200 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775106    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775401 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775309    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775401 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775357    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775516 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775477    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775570    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.775608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.775577    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177565c0-e6b6-4421-b395-986c4fc84d80-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.777307 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.777287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177565c0-e6b6-4421-b395-986c4fc84d80-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.777875 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.777854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177565c0-e6b6-4421-b395-986c4fc84d80-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.790731 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.790705    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dp8x\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-kube-api-access-2dp8x\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.792359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.792342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177565c0-e6b6-4421-b395-986c4fc84d80-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-7xtk4\" (UID: \"177565c0-e6b6-4421-b395-986c4fc84d80\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:16.890851 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:16.890832    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:17.023578 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:17.023485    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"]
May 20 15:45:17.025680 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:45:17.025654    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177565c0_e6b6_4421_b395_986c4fc84d80.slice/crio-9b237a0c677d637675e3c6dc8f4699db7b976fb0da4ffc0ac8bec50c1a6ee022 WatchSource:0}: Error finding container 9b237a0c677d637675e3c6dc8f4699db7b976fb0da4ffc0ac8bec50c1a6ee022: Status 404 returned error can't find the container with id 9b237a0c677d637675e3c6dc8f4699db7b976fb0da4ffc0ac8bec50c1a6ee022
May 20 15:45:17.071729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:17.071701    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4" event={"ID":"177565c0-e6b6-4421-b395-986c4fc84d80","Type":"ContainerStarted","Data":"9b237a0c677d637675e3c6dc8f4699db7b976fb0da4ffc0ac8bec50c1a6ee022"}
May 20 15:45:20.688565 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:20.688529    2574 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
May 20 15:45:20.688848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:20.688599    2574 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
May 20 15:45:20.688848 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:20.688631    2574 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
May 20 15:45:21.084385 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:21.084355    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4" event={"ID":"177565c0-e6b6-4421-b395-986c4fc84d80","Type":"ContainerStarted","Data":"40ff20c866bd95f7684fd9c0bd6ea47d0061da83f6deabb0370863b357602ff6"}
May 20 15:45:21.107058 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:21.107015    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4" podStartSLOduration=1.446040343 podStartE2EDuration="5.107001714s" podCreationTimestamp="2026-05-20 15:45:16 +0000 UTC" firstStartedPulling="2026-05-20 15:45:17.027386086 +0000 UTC m=+815.768136742" lastFinishedPulling="2026-05-20 15:45:20.68834745 +0000 UTC m=+819.429098113" observedRunningTime="2026-05-20 15:45:21.105669323 +0000 UTC m=+819.846419998" watchObservedRunningTime="2026-05-20 15:45:21.107001714 +0000 UTC m=+819.847752391"
May 20 15:45:21.891304 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:21.891276    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:21.895556 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:21.895534    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:22.086719 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:22.086681    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:22.087482 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:22.087466    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-7xtk4"
May 20 15:45:29.998270 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:29.998235    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:45:30.003317 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.003297    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.006501 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.006477    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-td2tg\""
May 20 15:45:30.006635 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.006478    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:45:30.006635 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.006476    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 20 15:45:30.012871 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.012851    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:45:30.072775 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.072867 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.072867 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.072947 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.072947 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94r9\" (UniqueName: \"kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.073012 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.072970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174340 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174314    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174351    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174372    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m94r9\" (UniqueName: \"kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174405    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174434    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174459    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174747 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174727    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174852 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174904 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.174904 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.174876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.177069 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.177049    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.181978 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.181959    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94r9\" (UniqueName: \"kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.313455 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.313378    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:45:30.451594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:30.451562    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:45:30.453604 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:45:30.453581    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6587dc_fcb3_4787_bec5_d2a59f096ada.slice/crio-c5638f41410db6245947472492cdbf20e14b4a76383475efce3dd24152bfd3db WatchSource:0}: Error finding container c5638f41410db6245947472492cdbf20e14b4a76383475efce3dd24152bfd3db: Status 404 returned error can't find the container with id c5638f41410db6245947472492cdbf20e14b4a76383475efce3dd24152bfd3db
May 20 15:45:31.112257 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:31.112199    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerStarted","Data":"c5638f41410db6245947472492cdbf20e14b4a76383475efce3dd24152bfd3db"}
May 20 15:45:34.123631 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:34.123585    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerStarted","Data":"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"}
May 20 15:45:35.127133 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:35.127102    2574 generic.go:358] "Generic (PLEG): container finished" podID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerID="0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694" exitCode=0
May 20 15:45:35.127473 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:35.127168    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerDied","Data":"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"}
May 20 15:45:37.135264 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:45:37.135227    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerStarted","Data":"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"}
May 20 15:46:08.266015 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:08.265978    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerStarted","Data":"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"}
May 20 15:46:08.266484 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:08.266315    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:46:08.268937 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:08.268915    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:46:08.288258 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:08.288221    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" podStartSLOduration=2.1748463510000002 podStartE2EDuration="39.28820975s" podCreationTimestamp="2026-05-20 15:45:29 +0000 UTC" firstStartedPulling="2026-05-20 15:45:30.45536633 +0000 UTC m=+829.196116985" lastFinishedPulling="2026-05-20 15:46:07.568729724 +0000 UTC m=+866.309480384" observedRunningTime="2026-05-20 15:46:08.286930388 +0000 UTC m=+867.027681067" watchObservedRunningTime="2026-05-20 15:46:08.28820975 +0000 UTC m=+867.028960429"
May 20 15:46:10.314321 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:10.314284    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:46:10.314703 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:10.314335    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:46:12.628687 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.628653    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 15:46:12.635517 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.635491    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.638378 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.638347    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 20 15:46:12.641159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.641135    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 15:46:12.765912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.765864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.765912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.765899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.766194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.765926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.766194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.765997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.766194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.766032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszdk\" (UniqueName: \"kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.766194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.766061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.766194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.766139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867274 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867235    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867274 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867276    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867307    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867336    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867357    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867383    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lszdk\" (UniqueName: \"kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867507 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867421    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867772 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867674    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867772 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867753    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.867887 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867817    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.868001 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.867981    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.869649 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.869618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.869852 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.869832    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.875341 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.875316    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszdk\" (UniqueName: \"kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:12.949126 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:12.949065    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 15:46:13.112537 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:13.112389    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 15:46:13.135846 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:46:13.135789    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ecf680_56b3_4337_a901_0c7e981fb764.slice/crio-73464bba143e27c6ff547c31958c958e08718591e392b8af3d9af35d939a5f53 WatchSource:0}: Error finding container 73464bba143e27c6ff547c31958c958e08718591e392b8af3d9af35d939a5f53: Status 404 returned error can't find the container with id 73464bba143e27c6ff547c31958c958e08718591e392b8af3d9af35d939a5f53
May 20 15:46:13.141277 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:13.139683    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:46:13.281575 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:13.281549    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g" event={"ID":"43ecf680-56b3-4337-a901-0c7e981fb764","Type":"ContainerStarted","Data":"73464bba143e27c6ff547c31958c958e08718591e392b8af3d9af35d939a5f53"}
May 20 15:46:14.287617 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:14.287580    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g" event={"ID":"43ecf680-56b3-4337-a901-0c7e981fb764","Type":"ContainerStarted","Data":"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"}
May 20 15:46:20.315409 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:20.315376    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:46:20.316607 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:46:20.316587    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:47:41.819757 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:41.819725    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:47:41.820224 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:41.820008    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="main" containerID="cri-o://7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31" gracePeriod=30
May 20 15:47:41.820224 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:41.820047    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="tokenizer" containerID="cri-o://3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82" gracePeriod=30
May 20 15:47:42.526212 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:42.526170    2574 generic.go:358] "Generic (PLEG): container finished" podID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerID="7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31" exitCode=0
May 20 15:47:42.526380 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:42.526246    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerDied","Data":"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"}
May 20 15:47:42.958822 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:42.958800    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:47:43.054350 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054325    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054360    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054405    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m94r9\" (UniqueName: \"kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054440    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054521 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054483    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054728 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054602    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache\") pod \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\" (UID: \"cd6587dc-fcb3-4787-bec5-d2a59f096ada\") "
May 20 15:47:43.054787 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054731    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:47:43.054787 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054751    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:47:43.054928 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.054891    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:47:43.055032 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.055005    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.055032 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.055024    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.055152 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.055040    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.055195 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.055166    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:47:43.056649 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.056628    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:47:43.056716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.056691    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9" (OuterVolumeSpecName: "kube-api-access-m94r9") pod "cd6587dc-fcb3-4787-bec5-d2a59f096ada" (UID: "cd6587dc-fcb3-4787-bec5-d2a59f096ada"). InnerVolumeSpecName "kube-api-access-m94r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:47:43.155414 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.155353    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.155414 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.155383    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94r9\" (UniqueName: \"kubernetes.io/projected/cd6587dc-fcb3-4787-bec5-d2a59f096ada-kube-api-access-m94r9\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.155414 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.155393    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6587dc-fcb3-4787-bec5-d2a59f096ada-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:47:43.530594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.530531    2574 generic.go:358] "Generic (PLEG): container finished" podID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerID="3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82" exitCode=0
May 20 15:47:43.530594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.530563    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerDied","Data":"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"}
May 20 15:47:43.530594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.530583    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2" event={"ID":"cd6587dc-fcb3-4787-bec5-d2a59f096ada","Type":"ContainerDied","Data":"c5638f41410db6245947472492cdbf20e14b4a76383475efce3dd24152bfd3db"}
May 20 15:47:43.530769 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.530600    2574 scope.go:117] "RemoveContainer" containerID="3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"
May 20 15:47:43.530769 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.530598    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"
May 20 15:47:43.538448 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.538430    2574 scope.go:117] "RemoveContainer" containerID="7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"
May 20 15:47:43.544941 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.544927    2574 scope.go:117] "RemoveContainer" containerID="0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"
May 20 15:47:43.551422 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.551406    2574 scope.go:117] "RemoveContainer" containerID="3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"
May 20 15:47:43.551665 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:47:43.551648    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82\": container with ID starting with 3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82 not found: ID does not exist" containerID="3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"
May 20 15:47:43.551713 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.551674    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82"} err="failed to get container status \"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82\": rpc error: code = NotFound desc = could not find container \"3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82\": container with ID starting with 3468662a875c79cf0cc12e425877cdf9f7b6d44f65a7e871460a86aca3d64f82 not found: ID does not exist"
May 20 15:47:43.551713 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.551692    2574 scope.go:117] "RemoveContainer" containerID="7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"
May 20 15:47:43.551922 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:47:43.551903    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31\": container with ID starting with 7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31 not found: ID does not exist" containerID="7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"
May 20 15:47:43.551960 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.551928    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31"} err="failed to get container status \"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31\": rpc error: code = NotFound desc = could not find container \"7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31\": container with ID starting with 7242388fb7bbf887364c600ca53be06d076433ad0702f10c69f2f93f22a77b31 not found: ID does not exist"
May 20 15:47:43.551960 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.551946    2574 scope.go:117] "RemoveContainer" containerID="0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"
May 20 15:47:43.552179 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:47:43.552162    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694\": container with ID starting with 0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694 not found: ID does not exist" containerID="0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"
May 20 15:47:43.552241 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.552188    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694"} err="failed to get container status \"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694\": rpc error: code = NotFound desc = could not find container \"0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694\": container with ID starting with 0cbd4b500356b99636fd4ff97599abef976527084d88112edb4cb044ff79a694 not found: ID does not exist"
May 20 15:47:43.558846 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.558828    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:47:43.565557 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.565536    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-56bf4h68g2"]
May 20 15:47:43.861480 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:43.861454    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" path="/var/lib/kubelet/pods/cd6587dc-fcb3-4787-bec5-d2a59f096ada/volumes"
May 20 15:47:49.775559 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775525    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775786    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="main"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775797    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="main"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775812    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="storage-initializer"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775817    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="storage-initializer"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775827    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="tokenizer"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775832    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="tokenizer"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775883    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="main"
May 20 15:47:49.775912 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.775893    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd6587dc-fcb3-4787-bec5-d2a59f096ada" containerName="tokenizer"
May 20 15:47:49.778691 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.778673    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.781301 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.781277    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 20 15:47:49.781409 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.781302    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-f57z5\""
May 20 15:47:49.794967 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.794941    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:47:49.904724 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.904844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.904844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8td78\" (UniqueName: \"kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.904844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.904844 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:49.904995 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:49.904850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.005992 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.005962    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006124 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.005996    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006124 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006018    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006124 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006068    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006124 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006109    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006270 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006140    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8td78\" (UniqueName: \"kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006533 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006532    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006560    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.006598 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.006588    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.008419 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.008400    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.021387 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.021365    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8td78\" (UniqueName: \"kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.087808 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.087784    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:50.214497 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.214432    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:47:50.217049 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:47:50.217024    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd226e757_9d56_465c_9d3b_2a4bbcd6e21f.slice/crio-056a1932e5c38880248130c3c9491fdd8ab99c4409d5925de388d3fac4cc09aa WatchSource:0}: Error finding container 056a1932e5c38880248130c3c9491fdd8ab99c4409d5925de388d3fac4cc09aa: Status 404 returned error can't find the container with id 056a1932e5c38880248130c3c9491fdd8ab99c4409d5925de388d3fac4cc09aa
May 20 15:47:50.552569 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.552476    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerStarted","Data":"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"}
May 20 15:47:50.552569 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:50.552516    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerStarted","Data":"056a1932e5c38880248130c3c9491fdd8ab99c4409d5925de388d3fac4cc09aa"}
May 20 15:47:51.556299 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:51.556267    2574 generic.go:358] "Generic (PLEG): container finished" podID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerID="5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a" exitCode=0
May 20 15:47:51.556722 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:51.556364    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerDied","Data":"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"}
May 20 15:47:52.561293 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:52.561260    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerStarted","Data":"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"}
May 20 15:47:52.561293 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:52.561299    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerStarted","Data":"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"}
May 20 15:47:52.561805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:52.561397    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:47:52.596622 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:47:52.596578    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" podStartSLOduration=3.5965656040000002 podStartE2EDuration="3.596565604s" podCreationTimestamp="2026-05-20 15:47:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:47:52.595171262 +0000 UTC m=+971.335921941" watchObservedRunningTime="2026-05-20 15:47:52.596565604 +0000 UTC m=+971.337316291"
May 20 15:48:00.088827 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:48:00.088748    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:48:00.088827 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:48:00.088788    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:48:00.091559 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:48:00.091533    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:48:00.585567 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:48:00.585541    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:48:21.588387 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:48:21.588361    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:49:11.481194 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:11.481160    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:49:11.481684 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:11.481591    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="main" containerID="cri-o://8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91" gracePeriod=30
May 20 15:49:11.481684 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:11.481600    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="tokenizer" containerID="cri-o://1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead" gracePeriod=30
May 20 15:49:11.588306 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:49:11.588275    2574 logging.go:55] [core] [Channel #116 SubChannel #117]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.26:9003", ServerName: "10.133.0.26:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.26:9003: connect: connection refused"
May 20 15:49:11.771831 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:11.771747    2574 generic.go:358] "Generic (PLEG): container finished" podID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerID="8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91" exitCode=0
May 20 15:49:11.771975 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:11.771822    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerDied","Data":"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"}
May 20 15:49:12.588270 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.588225    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.26:9003\" within 1s: context deadline exceeded"
May 20 15:49:12.630348 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.630326    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:49:12.715545 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715478    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715656 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715576    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715656 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715616    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715766 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715681    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715766 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715752    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715879 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715782    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8td78\" (UniqueName: \"kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78\") pod \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\" (UID: \"d226e757-9d56-465c-9d3b-2a4bbcd6e21f\") "
May 20 15:49:12.715949 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715923    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:49:12.716005 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715962    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:49:12.716005 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.715974    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:49:12.716156 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.716138    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:12.716219 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.716162    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:12.716219 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.716176    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:12.716514 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.716492    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:49:12.717520 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.717498    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:49:12.717820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.717804    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78" (OuterVolumeSpecName: "kube-api-access-8td78") pod "d226e757-9d56-465c-9d3b-2a4bbcd6e21f" (UID: "d226e757-9d56-465c-9d3b-2a4bbcd6e21f"). InnerVolumeSpecName "kube-api-access-8td78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:49:12.775752 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.775727    2574 generic.go:358] "Generic (PLEG): container finished" podID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerID="1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead" exitCode=0
May 20 15:49:12.775847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.775757    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerDied","Data":"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"}
May 20 15:49:12.775847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.775777    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k" event={"ID":"d226e757-9d56-465c-9d3b-2a4bbcd6e21f","Type":"ContainerDied","Data":"056a1932e5c38880248130c3c9491fdd8ab99c4409d5925de388d3fac4cc09aa"}
May 20 15:49:12.775847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.775793    2574 scope.go:117] "RemoveContainer" containerID="1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"
May 20 15:49:12.775847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.775809    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"
May 20 15:49:12.789608 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.789592    2574 scope.go:117] "RemoveContainer" containerID="8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"
May 20 15:49:12.796106 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.796072    2574 scope.go:117] "RemoveContainer" containerID="5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"
May 20 15:49:12.802516 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.802494    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:49:12.803313 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803295    2574 scope.go:117] "RemoveContainer" containerID="1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"
May 20 15:49:12.803533 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:49:12.803517    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead\": container with ID starting with 1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead not found: ID does not exist" containerID="1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"
May 20 15:49:12.803579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803538    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead"} err="failed to get container status \"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead\": rpc error: code = NotFound desc = could not find container \"1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead\": container with ID starting with 1f22d0445d30a0a398a37a05f53b5642e01945905495d8b1c67d27dd50f0cead not found: ID does not exist"
May 20 15:49:12.803579 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803555    2574 scope.go:117] "RemoveContainer" containerID="8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"
May 20 15:49:12.803767 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:49:12.803747    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91\": container with ID starting with 8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91 not found: ID does not exist" containerID="8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"
May 20 15:49:12.803807 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803773    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91"} err="failed to get container status \"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91\": rpc error: code = NotFound desc = could not find container \"8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91\": container with ID starting with 8c2182c4cb9f94ee3d312964370cef52105e68dc535900d1f9ae039e8c7a8b91 not found: ID does not exist"
May 20 15:49:12.803807 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803789    2574 scope.go:117] "RemoveContainer" containerID="5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"
May 20 15:49:12.803979 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:49:12.803964    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a\": container with ID starting with 5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a not found: ID does not exist" containerID="5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"
May 20 15:49:12.804018 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.803982    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a"} err="failed to get container status \"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a\": rpc error: code = NotFound desc = could not find container \"5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a\": container with ID starting with 5c6f60c8de85092619dc21f226d29c012b35c3f5abee13d88e58c77bfd000e2a not found: ID does not exist"
May 20 15:49:12.806489 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.806471    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-854df7mv6k"]
May 20 15:49:12.816995 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.816977    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:12.817075 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.816998    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:12.817075 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:12.817014    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8td78\" (UniqueName: \"kubernetes.io/projected/d226e757-9d56-465c-9d3b-2a4bbcd6e21f-kube-api-access-8td78\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:49:13.861597 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:13.861563    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" path="/var/lib/kubelet/pods/d226e757-9d56-465c-9d3b-2a4bbcd6e21f/volumes"
May 20 15:49:24.707502 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707471    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:49:24.707882 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707865    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="storage-initializer"
May 20 15:49:24.707930 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707887    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="storage-initializer"
May 20 15:49:24.707930 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707905    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="main"
May 20 15:49:24.707930 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707913    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="main"
May 20 15:49:24.708055 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707936    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="tokenizer"
May 20 15:49:24.708055 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.707945    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="tokenizer"
May 20 15:49:24.708055 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.708007    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="tokenizer"
May 20 15:49:24.708055 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.708018    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="d226e757-9d56-465c-9d3b-2a4bbcd6e21f" containerName="main"
May 20 15:49:24.711363 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.711348    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.713920 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.713903    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-qvscn\""
May 20 15:49:24.714220 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.714204    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 20 15:49:24.732159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.732128    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:49:24.797677 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.797797 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.797797 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.797797 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.797934 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.797934 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.797843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbctq\" (UniqueName: \"kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898182 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898157    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898290 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898195    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898290 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898214    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898290 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898250    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898290 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898272    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbctq\" (UniqueName: \"kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898434 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898308    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898600 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898574    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898704 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898610    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898704 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898684    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.898781 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.898740    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.900609 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.900589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:24.906618 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:24.906594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbctq\" (UniqueName: \"kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:25.023625 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:25.023569    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:25.149734 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:25.149709    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:49:25.152176 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:49:25.152141    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab79aa2_cd66_4a48_b052_d4f8cce7e7b7.slice/crio-c830f9ccb5f1b2eb42fcc72d716a1ca59b553b2a47b87a5b8bc8a0d2df6199a2 WatchSource:0}: Error finding container c830f9ccb5f1b2eb42fcc72d716a1ca59b553b2a47b87a5b8bc8a0d2df6199a2: Status 404 returned error can't find the container with id c830f9ccb5f1b2eb42fcc72d716a1ca59b553b2a47b87a5b8bc8a0d2df6199a2
May 20 15:49:25.815636 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:25.815598    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerStarted","Data":"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"}
May 20 15:49:25.815636 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:25.815637    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerStarted","Data":"c830f9ccb5f1b2eb42fcc72d716a1ca59b553b2a47b87a5b8bc8a0d2df6199a2"}
May 20 15:49:26.819177 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:26.819140    2574 generic.go:358] "Generic (PLEG): container finished" podID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerID="884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c" exitCode=0
May 20 15:49:26.819523 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:26.819183    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerDied","Data":"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"}
May 20 15:49:27.824099 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:27.824061    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerStarted","Data":"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"}
May 20 15:49:27.824099 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:27.824106    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerStarted","Data":"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"}
May 20 15:49:27.824534 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:27.824191    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:27.847694 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:27.847643    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" podStartSLOduration=3.847625594 podStartE2EDuration="3.847625594s" podCreationTimestamp="2026-05-20 15:49:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:49:27.846114891 +0000 UTC m=+1066.586865581" watchObservedRunningTime="2026-05-20 15:49:27.847625594 +0000 UTC m=+1066.588376276"
May 20 15:49:35.024046 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:35.024016    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:35.024046 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:35.024049    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:35.026459 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:35.026434    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:35.846163 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:35.846127    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:49:56.849802 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:49:56.849776    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:51:06.539026 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:06.538934    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:51:06.539554 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:06.539353    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="main" containerID="cri-o://a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e" gracePeriod=30
May 20 15:51:06.539554 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:06.539387    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="tokenizer" containerID="cri-o://6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a" gracePeriod=30
May 20 15:51:06.848941 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:51:06.848912    2574 logging.go:55] [core] [Channel #169 SubChannel #170]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.27:9003", ServerName: "10.133.0.27:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.27:9003: connect: connection refused"
May 20 15:51:07.097729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.097647    2574 generic.go:358] "Generic (PLEG): container finished" podID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerID="a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e" exitCode=0
May 20 15:51:07.097908 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.097731    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerDied","Data":"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"}
May 20 15:51:07.599974 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.599954    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:51:07.741945 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.741882    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbctq\" (UniqueName: \"kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.741945 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.741916    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.741945 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.741934    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.742161 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.741964    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.742161 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742016    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.742161 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742042    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds\") pod \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\" (UID: \"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7\") "
May 20 15:51:07.742302 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742181    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:07.742302 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742296    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.742408 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742381    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:07.742460 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742420    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:07.742721 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.742704    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:07.744017 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.744000    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq" (OuterVolumeSpecName: "kube-api-access-lbctq") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "kube-api-access-lbctq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:51:07.744113 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.744042    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" (UID: "5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:51:07.843062 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.843040    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.843062 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.843060    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.843204 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.843075    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.843204 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.843099    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbctq\" (UniqueName: \"kubernetes.io/projected/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kube-api-access-lbctq\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.843204 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.843112    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:51:07.849051 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:07.849026    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.27:9003\" within 1s: context deadline exceeded"
May 20 15:51:08.101723 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.101688    2574 generic.go:358] "Generic (PLEG): container finished" podID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerID="6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a" exitCode=0
May 20 15:51:08.101880 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.101762    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerDied","Data":"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"}
May 20 15:51:08.101880 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.101780    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"
May 20 15:51:08.101880 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.101803    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk" event={"ID":"5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7","Type":"ContainerDied","Data":"c830f9ccb5f1b2eb42fcc72d716a1ca59b553b2a47b87a5b8bc8a0d2df6199a2"}
May 20 15:51:08.101880 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.101819    2574 scope.go:117] "RemoveContainer" containerID="6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"
May 20 15:51:08.109078 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.109063    2574 scope.go:117] "RemoveContainer" containerID="a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"
May 20 15:51:08.115691 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.115678    2574 scope.go:117] "RemoveContainer" containerID="884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"
May 20 15:51:08.120781 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.120761    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:51:08.122614 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.122596    2574 scope.go:117] "RemoveContainer" containerID="6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"
May 20 15:51:08.122858 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:51:08.122841    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a\": container with ID starting with 6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a not found: ID does not exist" containerID="6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"
May 20 15:51:08.122916 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.122866    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a"} err="failed to get container status \"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a\": rpc error: code = NotFound desc = could not find container \"6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a\": container with ID starting with 6aa526a8db63c8747a8c14b548906259b8c3c5f17a60997e7c980b4c77ff347a not found: ID does not exist"
May 20 15:51:08.122916 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.122885    2574 scope.go:117] "RemoveContainer" containerID="a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"
May 20 15:51:08.123150 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:51:08.123119    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e\": container with ID starting with a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e not found: ID does not exist" containerID="a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"
May 20 15:51:08.123509 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.123151    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e"} err="failed to get container status \"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e\": rpc error: code = NotFound desc = could not find container \"a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e\": container with ID starting with a4c94d962062527ba13a84b197f73681d394d1260cec12d91b8a4d77eddc1b6e not found: ID does not exist"
May 20 15:51:08.123509 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.123175    2574 scope.go:117] "RemoveContainer" containerID="884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"
May 20 15:51:08.123509 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:51:08.123414    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c\": container with ID starting with 884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c not found: ID does not exist" containerID="884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"
May 20 15:51:08.123509 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.123441    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c"} err="failed to get container status \"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c\": rpc error: code = NotFound desc = could not find container \"884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c\": container with ID starting with 884083745cfd1dfc369c8d62f72155c62150072f8ac90ddc8d1d05dcc680cb9c not found: ID does not exist"
May 20 15:51:08.124753 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:08.124733    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7fjrwk"]
May 20 15:51:09.861247 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:09.861216    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" path="/var/lib/kubelet/pods/5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7/volumes"
May 20 15:51:17.938719 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938689    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938945    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="main"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938956    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="main"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938964    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="tokenizer"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938970    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="tokenizer"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938978    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="storage-initializer"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.938983    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="storage-initializer"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.939057    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="tokenizer"
May 20 15:51:17.939180 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.939065    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="5ab79aa2-cd66-4a48-b052-d4f8cce7e7b7" containerName="main"
May 20 15:51:17.942838 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.942818    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:17.945406 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.945385    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 20 15:51:17.954253 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:17.954235    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:51:18.015921 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.015896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016037 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.015929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016037 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.015953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016037 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.016022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.016052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4fzh\" (UniqueName: \"kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.016111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.016159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.016140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.116701 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116670    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4fzh\" (UniqueName: \"kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.116860 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116721    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.116860 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116765    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.116860 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116814    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.116860 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116848    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117068 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116880    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117068 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.116919    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117297 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.117215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117297 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.117284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117395 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.117336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.117433 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.117404    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.119077 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.119057    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.119215 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.119199    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.127262 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.127237    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4fzh\" (UniqueName: \"kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh\") pod \"precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.253467 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.253404    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:51:18.375690 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.375667    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:51:18.378281 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:51:18.378253    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb56343_f6df_4028_a23b_514df0c5742c.slice/crio-43c1534dbe10d37c2559a3f7dec45672211e6472df774e202cd2d5e462c3d5e8 WatchSource:0}: Error finding container 43c1534dbe10d37c2559a3f7dec45672211e6472df774e202cd2d5e462c3d5e8: Status 404 returned error can't find the container with id 43c1534dbe10d37c2559a3f7dec45672211e6472df774e202cd2d5e462c3d5e8
May 20 15:51:18.379949 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:18.379930    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:51:19.133433 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:19.133391    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerStarted","Data":"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"}
May 20 15:51:19.133433 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:51:19.133438    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerStarted","Data":"43c1534dbe10d37c2559a3f7dec45672211e6472df774e202cd2d5e462c3d5e8"}
May 20 15:52:15.284534 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:15.284496    2574 generic.go:358] "Generic (PLEG): container finished" podID="3fb56343-f6df-4028-a23b-514df0c5742c" containerID="8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8" exitCode=0
May 20 15:52:15.284948 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:15.284566    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerDied","Data":"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"}
May 20 15:52:17.292750 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:17.292718    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerStarted","Data":"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"}
May 20 15:52:17.316256 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:17.316215    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" podStartSLOduration=59.059583579 podStartE2EDuration="1m0.316201442s" podCreationTimestamp="2026-05-20 15:51:17 +0000 UTC" firstStartedPulling="2026-05-20 15:52:15.285667929 +0000 UTC m=+1234.026418584" lastFinishedPulling="2026-05-20 15:52:16.542285792 +0000 UTC m=+1235.283036447" observedRunningTime="2026-05-20 15:52:17.314633235 +0000 UTC m=+1236.055383924" watchObservedRunningTime="2026-05-20 15:52:17.316201442 +0000 UTC m=+1236.056952119"
May 20 15:52:18.254291 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:18.254248    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:18.254291 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:18.254291    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:18.266699 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:18.266678    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:18.306904 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:18.306877    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:19.302776 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:19.302744    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:52:19.475702 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:19.475672    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs: secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:19.476065 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:19.475750    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs podName:3fb56343-f6df-4028-a23b-514df0c5742c nodeName:}" failed. No retries permitted until 2026-05-20 15:52:19.975728649 +0000 UTC m=+1238.716479310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs") pod "precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c") : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:19.980902 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:19.980872    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs: secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:19.981065 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:19.980932    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs podName:3fb56343-f6df-4028-a23b-514df0c5742c nodeName:}" failed. No retries permitted until 2026-05-20 15:52:20.980918133 +0000 UTC m=+1239.721668792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs") pod "precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c") : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:20.989465 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:20.989436    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs: secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:20.989910 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:20.989498    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs podName:3fb56343-f6df-4028-a23b-514df0c5742c nodeName:}" failed. No retries permitted until 2026-05-20 15:52:22.989482014 +0000 UTC m=+1241.730232672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs") pod "precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c") : secret "precise-prefix-cache-test-kserve-self-signed-certs" not found
May 20 15:52:21.303883 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.303806    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="main" containerID="cri-o://f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971" gracePeriod=30
May 20 15:52:21.550817 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.550796    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:21.694903 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.694878    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695056 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.694926    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695056 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.694965    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695056 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.694987    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4fzh\" (UniqueName: \"kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695056 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695010    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695056 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695025    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695358 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695079    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm\") pod \"3fb56343-f6df-4028-a23b-514df0c5742c\" (UID: \"3fb56343-f6df-4028-a23b-514df0c5742c\") "
May 20 15:52:21.695358 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695280    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.695358 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695336    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.695358 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.695351    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home" (OuterVolumeSpecName: "home") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.697108 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.697061    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:52:21.697197 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.697132    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm" (OuterVolumeSpecName: "dshm") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.697197 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.697141    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh" (OuterVolumeSpecName: "kube-api-access-f4fzh") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "kube-api-access-f4fzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:52:21.754511 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.754484    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fb56343-f6df-4028-a23b-514df0c5742c" (UID: "3fb56343-f6df-4028-a23b-514df0c5742c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.796600 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796581    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796600 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796599    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb56343-f6df-4028-a23b-514df0c5742c-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796609    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796618    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796626    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4fzh\" (UniqueName: \"kubernetes.io/projected/3fb56343-f6df-4028-a23b-514df0c5742c-kube-api-access-f4fzh\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796635    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:21.796716 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:21.796643    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb56343-f6df-4028-a23b-514df0c5742c-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:52:22.307342 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.307308    2574 generic.go:358] "Generic (PLEG): container finished" podID="3fb56343-f6df-4028-a23b-514df0c5742c" containerID="f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971" exitCode=0
May 20 15:52:22.307748 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.307397    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"
May 20 15:52:22.307748 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.307398    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerDied","Data":"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"}
May 20 15:52:22.307748 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.307447    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b" event={"ID":"3fb56343-f6df-4028-a23b-514df0c5742c","Type":"ContainerDied","Data":"43c1534dbe10d37c2559a3f7dec45672211e6472df774e202cd2d5e462c3d5e8"}
May 20 15:52:22.307748 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.307471    2574 scope.go:117] "RemoveContainer" containerID="f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"
May 20 15:52:22.315031 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.315015    2574 scope.go:117] "RemoveContainer" containerID="8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"
May 20 15:52:22.324845 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.324826    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:52:22.329064 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.329046    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7fbf589c6b-bqg2b"]
May 20 15:52:22.379505 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.379485    2574 scope.go:117] "RemoveContainer" containerID="f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"
May 20 15:52:22.379769 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:22.379748    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971\": container with ID starting with f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971 not found: ID does not exist" containerID="f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"
May 20 15:52:22.379817 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.379780    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971"} err="failed to get container status \"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971\": rpc error: code = NotFound desc = could not find container \"f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971\": container with ID starting with f8230c95bdf22be28ad10caab6f507fb210b104a2968378efe02b92dbcf22971 not found: ID does not exist"
May 20 15:52:22.379817 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.379797    2574 scope.go:117] "RemoveContainer" containerID="8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"
May 20 15:52:22.380019 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:52:22.380003    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8\": container with ID starting with 8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8 not found: ID does not exist" containerID="8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"
May 20 15:52:22.380060 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:22.380023    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8"} err="failed to get container status \"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8\": rpc error: code = NotFound desc = could not find container \"8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8\": container with ID starting with 8d15e203f03bf667b3c5c9db8dfd6a8d248b6b3a87cf8b34daa209095eded2e8 not found: ID does not exist"
May 20 15:52:23.861440 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:23.861401    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" path="/var/lib/kubelet/pods/3fb56343-f6df-4028-a23b-514df0c5742c/volumes"
May 20 15:52:33.315040 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.314964    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:52:33.315530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.315245    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="storage-initializer"
May 20 15:52:33.315530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.315256    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="storage-initializer"
May 20 15:52:33.315530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.315275    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="main"
May 20 15:52:33.315530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.315281    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="main"
May 20 15:52:33.315530 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.315350    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="3fb56343-f6df-4028-a23b-514df0c5742c" containerName="main"
May 20 15:52:33.317171 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.317155    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.319651 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.319629    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-jx55x\""
May 20 15:52:33.319824 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.319808    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 15:52:33.327829 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.327809    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:52:33.380042 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380013    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.380153 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.380153 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8mt\" (UniqueName: \"kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.380248 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.380248 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.380248 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.380245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480856 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480829    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8mt\" (UniqueName: \"kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480970 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480862    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480970 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480901    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480970 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480919    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480970 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480944    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.480970 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.480971    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.481358 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.481333    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.481430 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.481338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.481430 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.481385    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.481430 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.481408    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.483389 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.483371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.489496 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.489471    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8mt\" (UniqueName: \"kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.626135 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.626116    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:33.753459 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:33.753425    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:52:33.755670 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:52:33.755642    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71780e82_67df_403a_a072_72d1bb414f6c.slice/crio-c3d45e8041f4a2bae32a848c973cea45c8b1f73e8cfcf743aff64fb72f9e0413 WatchSource:0}: Error finding container c3d45e8041f4a2bae32a848c973cea45c8b1f73e8cfcf743aff64fb72f9e0413: Status 404 returned error can't find the container with id c3d45e8041f4a2bae32a848c973cea45c8b1f73e8cfcf743aff64fb72f9e0413
May 20 15:52:34.341843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:34.341808    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerStarted","Data":"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"}
May 20 15:52:34.341843 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:34.341845    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerStarted","Data":"c3d45e8041f4a2bae32a848c973cea45c8b1f73e8cfcf743aff64fb72f9e0413"}
May 20 15:52:35.345506 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:35.345466    2574 generic.go:358] "Generic (PLEG): container finished" podID="71780e82-67df-403a-a072-72d1bb414f6c" containerID="ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827" exitCode=0
May 20 15:52:35.345879 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:35.345555    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerDied","Data":"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"}
May 20 15:52:36.350595 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:36.350550    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerStarted","Data":"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"}
May 20 15:52:36.350595 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:36.350590    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerStarted","Data":"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"}
May 20 15:52:36.351001 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:36.350685    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:36.373704 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:36.373657    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" podStartSLOduration=3.373639835 podStartE2EDuration="3.373639835s" podCreationTimestamp="2026-05-20 15:52:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:52:36.371023121 +0000 UTC m=+1255.111773799" watchObservedRunningTime="2026-05-20 15:52:36.373639835 +0000 UTC m=+1255.114390516"
May 20 15:52:43.626869 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:43.626838    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:43.627289 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:43.626987    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:43.629474 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:43.629450    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:52:44.375204 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:52:44.375179    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:53:06.382156 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:53:06.382128    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:54:54.213837 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.213804    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:54:54.214328 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.214128    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="main" containerID="cri-o://5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca" gracePeriod=30
May 20 15:54:54.214328 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.214175    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="tokenizer" containerID="cri-o://b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1" gracePeriod=30
May 20 15:54:54.374866 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.374829    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.29:8082/healthz\": dial tcp 10.133.0.29:8082: connect: connection refused"
May 20 15:54:54.721411 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.721380    2574 generic.go:358] "Generic (PLEG): container finished" podID="71780e82-67df-403a-a072-72d1bb414f6c" containerID="5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca" exitCode=0
May 20 15:54:54.721577 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:54.721437    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerDied","Data":"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"}
May 20 15:54:55.268183 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.268162    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:54:55.420796 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420707    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.420796 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420769    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.420796 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420803    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.421027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420831    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.421027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420870    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8mt\" (UniqueName: \"kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.421027 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.420893    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds\") pod \"71780e82-67df-403a-a072-72d1bb414f6c\" (UID: \"71780e82-67df-403a-a072-72d1bb414f6c\") "
May 20 15:54:55.421206 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.421110    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:54:55.421206 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.421120    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:54:55.421345 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.421322    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:54:55.421577 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.421556    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:54:55.422925 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.422903    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:54:55.423054 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.423031    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt" (OuterVolumeSpecName: "kube-api-access-8j8mt") pod "71780e82-67df-403a-a072-72d1bb414f6c" (UID: "71780e82-67df-403a-a072-72d1bb414f6c"). InnerVolumeSpecName "kube-api-access-8j8mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:54:55.521649 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521622    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.521649 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521647    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71780e82-67df-403a-a072-72d1bb414f6c-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.521805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521656    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8mt\" (UniqueName: \"kubernetes.io/projected/71780e82-67df-403a-a072-72d1bb414f6c-kube-api-access-8j8mt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.521805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521666    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.521805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521674    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.521805 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.521683    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71780e82-67df-403a-a072-72d1bb414f6c-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:54:55.725315 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.725238    2574 generic.go:358] "Generic (PLEG): container finished" podID="71780e82-67df-403a-a072-72d1bb414f6c" containerID="b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1" exitCode=0
May 20 15:54:55.725315 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.725278    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerDied","Data":"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"}
May 20 15:54:55.725315 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.725303    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb" event={"ID":"71780e82-67df-403a-a072-72d1bb414f6c","Type":"ContainerDied","Data":"c3d45e8041f4a2bae32a848c973cea45c8b1f73e8cfcf743aff64fb72f9e0413"}
May 20 15:54:55.725315 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.725311    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"
May 20 15:54:55.725544 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.725317    2574 scope.go:117] "RemoveContainer" containerID="b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"
May 20 15:54:55.733718 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.733702    2574 scope.go:117] "RemoveContainer" containerID="5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"
May 20 15:54:55.740340 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.740324    2574 scope.go:117] "RemoveContainer" containerID="ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"
May 20 15:54:55.749199 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.747699    2574 scope.go:117] "RemoveContainer" containerID="b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"
May 20 15:54:55.749823 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:54:55.749797    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1\": container with ID starting with b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1 not found: ID does not exist" containerID="b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"
May 20 15:54:55.749926 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.749834    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1"} err="failed to get container status \"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1\": rpc error: code = NotFound desc = could not find container \"b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1\": container with ID starting with b05f633a4a5020b8479ac3bf54df44376f358a1059d80f04d26444b940a833b1 not found: ID does not exist"
May 20 15:54:55.749926 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.749859    2574 scope.go:117] "RemoveContainer" containerID="5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"
May 20 15:54:55.750295 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:54:55.750258    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca\": container with ID starting with 5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca not found: ID does not exist" containerID="5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"
May 20 15:54:55.750390 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.750306    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca"} err="failed to get container status \"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca\": rpc error: code = NotFound desc = could not find container \"5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca\": container with ID starting with 5afafdf224a4cc834363692cef64df0eb27e73cde177232ae37a38ce71460cca not found: ID does not exist"
May 20 15:54:55.750390 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.750329    2574 scope.go:117] "RemoveContainer" containerID="ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"
May 20 15:54:55.750638 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:54:55.750610    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827\": container with ID starting with ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827 not found: ID does not exist" containerID="ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"
May 20 15:54:55.750773 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.750643    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827"} err="failed to get container status \"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827\": rpc error: code = NotFound desc = could not find container \"ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827\": container with ID starting with ffd5a0c777d8b5f7602a97640a8605dd2d02717bb9d4588c58238d55b6e5f827 not found: ID does not exist"
May 20 15:54:55.750866 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.750846    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:54:55.754748 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.754727    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-hjqtb"]
May 20 15:54:55.860993 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:55.860966    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71780e82-67df-403a-a072-72d1bb414f6c" path="/var/lib/kubelet/pods/71780e82-67df-403a-a072-72d1bb414f6c/volumes"
May 20 15:54:59.765706 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765674    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765955    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="storage-initializer"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765966    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="storage-initializer"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765975    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="tokenizer"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765980    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="tokenizer"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765990    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="main"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.765996    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="main"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.766050    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="main"
May 20 15:54:59.766162 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.766059    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="71780e82-67df-403a-a072-72d1bb414f6c" containerName="tokenizer"
May 20 15:54:59.770767 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.770751    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.773335 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.773302    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-5nr8w\""
May 20 15:54:59.773464 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.773353    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 15:54:59.779900 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.779876    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:54:59.948787 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.948919 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.948919 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.948919 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.949033 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:54:59.949033 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:54:59.948955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbwq\" (UniqueName: \"kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050325 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050238    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050325 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050286    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050325 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050319    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbwq\" (UniqueName: \"kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050359    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050391    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050563 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050450    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050686 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050672    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050768 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050830 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050783    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.050887 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.050865    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.052673 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.052656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.058918 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.058894    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbwq\" (UniqueName: \"kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq\") pod \"stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.080734 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.080714    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:00.210394 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.210368    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:55:00.212382 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:55:00.212350    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8454c5_a764_4f4e_94ab_d256edcec85e.slice/crio-e089570b6229de41e9275339ec44e7093fe0f3fed6c33b92d77462ac09bf6653 WatchSource:0}: Error finding container e089570b6229de41e9275339ec44e7093fe0f3fed6c33b92d77462ac09bf6653: Status 404 returned error can't find the container with id e089570b6229de41e9275339ec44e7093fe0f3fed6c33b92d77462ac09bf6653
May 20 15:55:00.741566 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.741527    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerStarted","Data":"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"}
May 20 15:55:00.741566 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:00.741564    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerStarted","Data":"e089570b6229de41e9275339ec44e7093fe0f3fed6c33b92d77462ac09bf6653"}
May 20 15:55:01.745814 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:01.745781    2574 generic.go:358] "Generic (PLEG): container finished" podID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerID="0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc" exitCode=0
May 20 15:55:01.746190 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:01.745823    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerDied","Data":"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"}
May 20 15:55:02.756144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:02.751118    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerStarted","Data":"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"}
May 20 15:55:02.756144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:02.751164    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerStarted","Data":"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"}
May 20 15:55:02.756144 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:02.752126    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:02.778379 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:02.778330    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" podStartSLOduration=3.778319087 podStartE2EDuration="3.778319087s" podCreationTimestamp="2026-05-20 15:54:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:55:02.776898122 +0000 UTC m=+1401.517648807" watchObservedRunningTime="2026-05-20 15:55:02.778319087 +0000 UTC m=+1401.519069765"
May 20 15:55:10.081718 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:10.081689    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:10.081718 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:10.081726    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:10.084197 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:10.084174    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:10.777016 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:10.776988    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:55:31.780487 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:55:31.780407    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:58:11.072131 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:11.072079    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:58:11.074618 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:11.072478    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="main" containerID="cri-o://b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6" gracePeriod=30
May 20 15:58:11.074618 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:11.072535    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="tokenizer" containerID="cri-o://b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c" gracePeriod=30
May 20 15:58:11.258223 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:11.258185    2574 generic.go:358] "Generic (PLEG): container finished" podID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerID="b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6" exitCode=0
May 20 15:58:11.258410 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:11.258233    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerDied","Data":"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"}
May 20 15:58:11.779432 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:58:11.779404    2574 logging.go:55] [core] [Channel #351 SubChannel #352]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.30:9003", ServerName: "10.133.0.30:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.30:9003: connect: connection refused"
May 20 15:58:12.215659 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.215639    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:58:12.262492 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.262462    2574 generic.go:358] "Generic (PLEG): container finished" podID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerID="b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c" exitCode=0
May 20 15:58:12.262594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.262532    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerDied","Data":"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"}
May 20 15:58:12.262594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.262535    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"
May 20 15:58:12.262594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.262554    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" event={"ID":"2d8454c5-a764-4f4e-94ab-d256edcec85e","Type":"ContainerDied","Data":"e089570b6229de41e9275339ec44e7093fe0f3fed6c33b92d77462ac09bf6653"}
May 20 15:58:12.262594 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.262574    2574 scope.go:117] "RemoveContainer" containerID="b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"
May 20 15:58:12.271215 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.271201    2574 scope.go:117] "RemoveContainer" containerID="b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"
May 20 15:58:12.278402 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.278386    2574 scope.go:117] "RemoveContainer" containerID="0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"
May 20 15:58:12.286212 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.286197    2574 scope.go:117] "RemoveContainer" containerID="b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"
May 20 15:58:12.286453 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:58:12.286438    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c\": container with ID starting with b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c not found: ID does not exist" containerID="b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"
May 20 15:58:12.286495 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.286462    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c"} err="failed to get container status \"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c\": rpc error: code = NotFound desc = could not find container \"b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c\": container with ID starting with b4f5e2961994eb849afae764b4b415ddd86b6ccab4ab2b74e0f477996085f62c not found: ID does not exist"
May 20 15:58:12.286495 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.286477    2574 scope.go:117] "RemoveContainer" containerID="b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"
May 20 15:58:12.286727 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:58:12.286705    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6\": container with ID starting with b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6 not found: ID does not exist" containerID="b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"
May 20 15:58:12.286825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.286730    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6"} err="failed to get container status \"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6\": rpc error: code = NotFound desc = could not find container \"b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6\": container with ID starting with b5ecd4c5cc7fc00a5d351aca70beadefc0952166f456db5bc437a5c123e257b6 not found: ID does not exist"
May 20 15:58:12.286825 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.286746    2574 scope.go:117] "RemoveContainer" containerID="0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"
May 20 15:58:12.287002 ip-10-0-141-222 kubenswrapper[2574]: E0520 15:58:12.286987    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc\": container with ID starting with 0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc not found: ID does not exist" containerID="0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"
May 20 15:58:12.287044 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.287006    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc"} err="failed to get container status \"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc\": rpc error: code = NotFound desc = could not find container \"0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc\": container with ID starting with 0f8bce879049fcd8926e6a0f2f116a64eec4ac0e07d4b7f62e4b3a72403c98bc not found: ID does not exist"
May 20 15:58:12.328958 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.328935    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329043 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.328985    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329043 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329031    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329174 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329113    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlbwq\" (UniqueName: \"kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329174 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329153    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329284 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329181    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache\") pod \"2d8454c5-a764-4f4e-94ab-d256edcec85e\" (UID: \"2d8454c5-a764-4f4e-94ab-d256edcec85e\") "
May 20 15:58:12.329344 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329273    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:12.329486 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329462    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:12.329486 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329477    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.329654 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329488    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:12.329799 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.329777    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:12.331052 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.331029    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:58:12.331146 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.331062    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq" (OuterVolumeSpecName: "kube-api-access-wlbwq") pod "2d8454c5-a764-4f4e-94ab-d256edcec85e" (UID: "2d8454c5-a764-4f4e-94ab-d256edcec85e"). InnerVolumeSpecName "kube-api-access-wlbwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:58:12.430585 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.430538    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbwq\" (UniqueName: \"kubernetes.io/projected/2d8454c5-a764-4f4e-94ab-d256edcec85e-kube-api-access-wlbwq\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.430585 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.430557    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.430585 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.430566    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.430585 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.430576    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8454c5-a764-4f4e-94ab-d256edcec85e-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.430585 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.430585    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d8454c5-a764-4f4e-94ab-d256edcec85e-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 15:58:12.584914 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.584879    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:58:12.588773 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.588750    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc"]
May 20 15:58:12.779438 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:12.779360    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6b77bc7d75-6wtmc" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.30:9003\" within 1s: context deadline exceeded"
May 20 15:58:13.861550 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:58:13.861518    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" path="/var/lib/kubelet/pods/2d8454c5-a764-4f4e-94ab-d256edcec85e/volumes"
May 20 15:59:26.188079 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188049    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188331    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="tokenizer"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188343    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="tokenizer"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188353    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="storage-initializer"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188359    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="storage-initializer"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188366    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="main"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188372    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="main"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188418    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="tokenizer"
May 20 15:59:26.188518 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.188427    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d8454c5-a764-4f4e-94ab-d256edcec85e" containerName="main"
May 20 15:59:26.191332 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.191315    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.194620 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.194599    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-5v9zs\""
May 20 15:59:26.195371 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.195343    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 20 15:59:26.202576 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.202555    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 15:59:26.238511 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6jk\" (UniqueName: \"kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.238597 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.238661 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.238729 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.238790 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.238847 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.238783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.339906 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.339883    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.339997 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.339916    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.339997 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.339934    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.339997 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.339952    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340188 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.339998    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6jk\" (UniqueName: \"kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340188 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.340044    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340322 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.340299    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340383 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.340342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340383 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.340371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.340450 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.340422    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.342264 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.342247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.348958 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.348938    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6jk\" (UniqueName: \"kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.500200 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.500124    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:26.627957 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.627927    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 15:59:26.630244 ip-10-0-141-222 kubenswrapper[2574]: W0520 15:59:26.630217    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38b5523_4b86_4537_b31f_d3a6b68e5da1.slice/crio-bca09ee98c9c1b41250e65b4de3e0027bcda2ab760c105e621afc8627595d6d7 WatchSource:0}: Error finding container bca09ee98c9c1b41250e65b4de3e0027bcda2ab760c105e621afc8627595d6d7: Status 404 returned error can't find the container with id bca09ee98c9c1b41250e65b4de3e0027bcda2ab760c105e621afc8627595d6d7
May 20 15:59:26.632159 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:26.632139    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:59:27.474305 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:27.474275    2574 generic.go:358] "Generic (PLEG): container finished" podID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerID="2ecc8d47042144dbb18978828cf8da9a9907e96af5fa13efd84eebbbfa65c852" exitCode=0
May 20 15:59:27.474639 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:27.474362    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerDied","Data":"2ecc8d47042144dbb18978828cf8da9a9907e96af5fa13efd84eebbbfa65c852"}
May 20 15:59:27.474639 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:27.474394    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerStarted","Data":"bca09ee98c9c1b41250e65b4de3e0027bcda2ab760c105e621afc8627595d6d7"}
May 20 15:59:28.479820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:28.479781    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerStarted","Data":"b2d64bf25e4ced2667bb0a74f426d30e7a20e159a980211fafbce9ef0b2f6aea"}
May 20 15:59:28.479820 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:28.479818    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerStarted","Data":"30cc6c2563b79eb6d8c3fb446df88a1880f2baf46ff477eb2a3cb0157812fa08"}
May 20 15:59:28.480359 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:28.479880    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:28.502311 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:28.502266    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" podStartSLOduration=2.502249859 podStartE2EDuration="2.502249859s" podCreationTimestamp="2026-05-20 15:59:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:59:28.499807389 +0000 UTC m=+1667.240558067" watchObservedRunningTime="2026-05-20 15:59:28.502249859 +0000 UTC m=+1667.243000539"
May 20 15:59:36.501222 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:36.501186    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:36.501795 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:36.501233    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:36.503968 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:36.503947    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:37.507368 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:37.507344    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 15:59:58.510050 ip-10-0-141-222 kubenswrapper[2574]: I0520 15:59:58.509983    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 16:01:02.240783 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:02.240753    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 16:01:02.241282 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:02.241022    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g" podUID="43ecf680-56b3-4337-a901-0c7e981fb764" containerName="storage-initializer" containerID="cri-o://d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2" gracePeriod=30
May 20 16:01:22.425905 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.425872    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:01:22.429191 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.429170    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.431590 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.431572    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-l9pwk\""
May 20 16:01:22.431795 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.431782    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 20 16:01:22.441794 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.441767    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:01:22.525407 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.525514 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6zc\" (UniqueName: \"kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.525514 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.525603 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.525603 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.525603 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.525568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.625921 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.625898    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626117 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.625926    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626117 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.625950    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626117 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.625977    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6zc\" (UniqueName: \"kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626117 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626008    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626117 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626035    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626389 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626406    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626489 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626467    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.626533 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.626496    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.628372 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.628352    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.636458 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.636431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6zc\" (UniqueName: \"kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.738800 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.738745    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:22.866026 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:22.866000    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:01:22.868877 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:01:22.868849    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d10382_4a9e_4f67_a61a_a19ca6c4ac44.slice/crio-f04769761dc260dc9eeb58e26f8a9d42638e1e06d4b78ab294d8ee73d00bbae0 WatchSource:0}: Error finding container f04769761dc260dc9eeb58e26f8a9d42638e1e06d4b78ab294d8ee73d00bbae0: Status 404 returned error can't find the container with id f04769761dc260dc9eeb58e26f8a9d42638e1e06d4b78ab294d8ee73d00bbae0
May 20 16:01:23.794288 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:23.794259    2574 generic.go:358] "Generic (PLEG): container finished" podID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerID="64b496144d27ec5e6a8ab69ccbf74b1db9b106a1545e4501f23690b89be45f72" exitCode=0
May 20 16:01:23.794600 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:23.794332    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerDied","Data":"64b496144d27ec5e6a8ab69ccbf74b1db9b106a1545e4501f23690b89be45f72"}
May 20 16:01:23.794600 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:23.794354    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerStarted","Data":"f04769761dc260dc9eeb58e26f8a9d42638e1e06d4b78ab294d8ee73d00bbae0"}
May 20 16:01:24.798977 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:24.798939    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerStarted","Data":"2425d52a59d546b1d0a2c7a31544305c573dd63f5c26e95229c3ea95f4a92671"}
May 20 16:01:24.798977 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:24.798976    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerStarted","Data":"03362024598ad68896a25f6430a971da526dfacdf675b707eb33bf38a9243b1b"}
May 20 16:01:24.799418 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:24.799079    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:24.826887 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:24.826841    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" podStartSLOduration=2.8268276759999997 podStartE2EDuration="2.826827676s" podCreationTimestamp="2026-05-20 16:01:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:01:24.825606442 +0000 UTC m=+1783.566357121" watchObservedRunningTime="2026-05-20 16:01:24.826827676 +0000 UTC m=+1783.567578357"
May 20 16:01:32.421676 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.421653    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g_43ecf680-56b3-4337-a901-0c7e981fb764/storage-initializer/0.log"
May 20 16:01:32.421955 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.421717    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 16:01:32.498950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.498869    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.498950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.498918    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.498950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.498943    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lszdk\" (UniqueName: \"kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.499236 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.498989    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.499236 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499029    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.499236 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499170    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.499378 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499217    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home" (OuterVolumeSpecName: "home") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:01:32.499378 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499247    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir\") pod \"43ecf680-56b3-4337-a901-0c7e981fb764\" (UID: \"43ecf680-56b3-4337-a901-0c7e981fb764\") "
May 20 16:01:32.499474 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499450    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache" (OuterVolumeSpecName: "model-cache") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:01:32.499474 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499462    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:01:32.499564 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499534    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.499564 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499554    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.499645 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.499569    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.501223 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.501196    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:01:32.501322 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.501216    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk" (OuterVolumeSpecName: "kube-api-access-lszdk") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "kube-api-access-lszdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:01:32.501371 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.501318    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm" (OuterVolumeSpecName: "dshm") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:01:32.518539 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.518514    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43ecf680-56b3-4337-a901-0c7e981fb764" (UID: "43ecf680-56b3-4337-a901-0c7e981fb764"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:01:32.600777 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.600752    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.600777 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.600778    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lszdk\" (UniqueName: \"kubernetes.io/projected/43ecf680-56b3-4337-a901-0c7e981fb764-kube-api-access-lszdk\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.600942 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.600790    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ecf680-56b3-4337-a901-0c7e981fb764-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.600942 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.600800    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ecf680-56b3-4337-a901-0c7e981fb764-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:01:32.739164 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.739129    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:32.739293 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.739181    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:32.741864 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.741841    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:32.822010 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.821941    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g_43ecf680-56b3-4337-a901-0c7e981fb764/storage-initializer/0.log"
May 20 16:01:32.822010 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.821977    2574 generic.go:358] "Generic (PLEG): container finished" podID="43ecf680-56b3-4337-a901-0c7e981fb764" containerID="d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2" exitCode=137
May 20 16:01:32.822200 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.822051    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"
May 20 16:01:32.822200 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.822069    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g" event={"ID":"43ecf680-56b3-4337-a901-0c7e981fb764","Type":"ContainerDied","Data":"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"}
May 20 16:01:32.822200 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.822117    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g" event={"ID":"43ecf680-56b3-4337-a901-0c7e981fb764","Type":"ContainerDied","Data":"73464bba143e27c6ff547c31958c958e08718591e392b8af3d9af35d939a5f53"}
May 20 16:01:32.822200 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.822133    2574 scope.go:117] "RemoveContainer" containerID="d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"
May 20 16:01:32.823556 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.823532    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:01:32.850841 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.850823    2574 scope.go:117] "RemoveContainer" containerID="d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"
May 20 16:01:32.851178 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:01:32.851130    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2\": container with ID starting with d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2 not found: ID does not exist" containerID="d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"
May 20 16:01:32.851178 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.851163    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2"} err="failed to get container status \"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2\": rpc error: code = NotFound desc = could not find container \"d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2\": container with ID starting with d26ce31618b64652ff113459272d0d0f6616875202679aa9640be71e025547e2 not found: ID does not exist"
May 20 16:01:32.876360 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.876339    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 16:01:32.880000 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:32.879976    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6677cbf748qw22g"]
May 20 16:01:33.861218 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:33.861187    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ecf680-56b3-4337-a901-0c7e981fb764" path="/var/lib/kubelet/pods/43ecf680-56b3-4337-a901-0c7e981fb764/volumes"
May 20 16:01:53.826835 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:01:53.826808    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:03:19.062485 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:19.062451    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:03:19.064052 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:19.062788    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="main" containerID="cri-o://03362024598ad68896a25f6430a971da526dfacdf675b707eb33bf38a9243b1b" gracePeriod=30
May 20 16:03:19.064052 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:19.062861    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="tokenizer" containerID="cri-o://2425d52a59d546b1d0a2c7a31544305c573dd63f5c26e95229c3ea95f4a92671" gracePeriod=30
May 20 16:03:20.134999 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.134961    2574 generic.go:358] "Generic (PLEG): container finished" podID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerID="2425d52a59d546b1d0a2c7a31544305c573dd63f5c26e95229c3ea95f4a92671" exitCode=0
May 20 16:03:20.134999 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.134998    2574 generic.go:358] "Generic (PLEG): container finished" podID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerID="03362024598ad68896a25f6430a971da526dfacdf675b707eb33bf38a9243b1b" exitCode=0
May 20 16:03:20.135382 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.135032    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerDied","Data":"2425d52a59d546b1d0a2c7a31544305c573dd63f5c26e95229c3ea95f4a92671"}
May 20 16:03:20.135382 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.135093    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerDied","Data":"03362024598ad68896a25f6430a971da526dfacdf675b707eb33bf38a9243b1b"}
May 20 16:03:20.212711 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.212681    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:03:20.260974 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.260940    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261150 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.260984    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb6zc\" (UniqueName: \"kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261150 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261040    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261150 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261070    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261150 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261143    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261375 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261200    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds\") pod \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\" (UID: \"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44\") "
May 20 16:03:20.261453 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261425    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:20.261512 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261481    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:20.261557 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261521    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:20.261807 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.261788    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:20.263104 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.263071    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:03:20.263254 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.263235    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc" (OuterVolumeSpecName: "kube-api-access-tb6zc") pod "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" (UID: "f2d10382-4a9e-4f67-a61a-a19ca6c4ac44"). InnerVolumeSpecName "kube-api-access-tb6zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:03:20.362417 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362381    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:20.362417 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362407    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:20.362417 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362417    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6zc\" (UniqueName: \"kubernetes.io/projected/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kube-api-access-tb6zc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:20.362648 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362431    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:20.362648 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362444    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:20.362648 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:20.362457    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:03:21.139617 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.139578    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf" event={"ID":"f2d10382-4a9e-4f67-a61a-a19ca6c4ac44","Type":"ContainerDied","Data":"f04769761dc260dc9eeb58e26f8a9d42638e1e06d4b78ab294d8ee73d00bbae0"}
May 20 16:03:21.139617 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.139607    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"
May 20 16:03:21.140125 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.139636    2574 scope.go:117] "RemoveContainer" containerID="2425d52a59d546b1d0a2c7a31544305c573dd63f5c26e95229c3ea95f4a92671"
May 20 16:03:21.147971 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.147954    2574 scope.go:117] "RemoveContainer" containerID="03362024598ad68896a25f6430a971da526dfacdf675b707eb33bf38a9243b1b"
May 20 16:03:21.154813 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.154797    2574 scope.go:117] "RemoveContainer" containerID="64b496144d27ec5e6a8ab69ccbf74b1db9b106a1545e4501f23690b89be45f72"
May 20 16:03:21.162133 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.162107    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:03:21.164519 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.164493    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheqqkkf"]
May 20 16:03:21.861520 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:21.861487    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" path="/var/lib/kubelet/pods/f2d10382-4a9e-4f67-a61a-a19ca6c4ac44/volumes"
May 20 16:03:28.141744 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.141711    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:03:28.142146 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142130    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="tokenizer"
May 20 16:03:28.142204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142149    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="tokenizer"
May 20 16:03:28.142204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142170    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="storage-initializer"
May 20 16:03:28.142204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142179    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="storage-initializer"
May 20 16:03:28.142204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142190    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="main"
May 20 16:03:28.142204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142198    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="main"
May 20 16:03:28.142351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142209    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ecf680-56b3-4337-a901-0c7e981fb764" containerName="storage-initializer"
May 20 16:03:28.142351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142218    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ecf680-56b3-4337-a901-0c7e981fb764" containerName="storage-initializer"
May 20 16:03:28.142351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142295    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="43ecf680-56b3-4337-a901-0c7e981fb764" containerName="storage-initializer"
May 20 16:03:28.142351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142306    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="main"
May 20 16:03:28.142351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.142315    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2d10382-4a9e-4f67-a61a-a19ca6c4ac44" containerName="tokenizer"
May 20 16:03:28.145711 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.145690    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.148078 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.148057    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-f6xjs\""
May 20 16:03:28.148348 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.148333    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 20 16:03:28.154033 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.154011    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:03:28.229474 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.229638 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.229638 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwp8\" (UniqueName: \"kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.229638 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.229757 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.229757 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.229681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.330979 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.330943    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331165 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.330986    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331165 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331049    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwp8\" (UniqueName: \"kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331165 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331131    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331165 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331160    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331370 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331192    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331442 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331419    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331513 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331492    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331560 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331544    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.331599 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.331555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.333518 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.333500    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.338775 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.338755    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwp8\" (UniqueName: \"kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8\") pod \"custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.456153 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.456054    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:28.587563 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:28.587538    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:03:28.590317 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:03:28.590279    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef890ab7_b1bd_4c30_bb08_01264d25b4df.slice/crio-52757ea9f3ce78ef27ca239f39163e1a510f3ce7894f0375fb6f7961ca7d1b75 WatchSource:0}: Error finding container 52757ea9f3ce78ef27ca239f39163e1a510f3ce7894f0375fb6f7961ca7d1b75: Status 404 returned error can't find the container with id 52757ea9f3ce78ef27ca239f39163e1a510f3ce7894f0375fb6f7961ca7d1b75
May 20 16:03:29.165560 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:29.165526    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerStarted","Data":"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"}
May 20 16:03:29.165952 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:29.165567    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerStarted","Data":"52757ea9f3ce78ef27ca239f39163e1a510f3ce7894f0375fb6f7961ca7d1b75"}
May 20 16:03:30.169305 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:30.169267    2574 generic.go:358] "Generic (PLEG): container finished" podID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerID="c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123" exitCode=0
May 20 16:03:30.169741 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:30.169354    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerDied","Data":"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"}
May 20 16:03:31.175237 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:31.175202    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerStarted","Data":"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"}
May 20 16:03:31.175237 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:31.175239    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerStarted","Data":"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"}
May 20 16:03:31.175677 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:31.175319    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:31.197242 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:31.197199    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" podStartSLOduration=3.19718592 podStartE2EDuration="3.19718592s" podCreationTimestamp="2026-05-20 16:03:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:03:31.195535962 +0000 UTC m=+1909.936286640" watchObservedRunningTime="2026-05-20 16:03:31.19718592 +0000 UTC m=+1909.937936597"
May 20 16:03:38.456644 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:38.456612    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:38.456644 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:38.456649    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:38.459158 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:38.459133    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:03:39.200996 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:03:39.200972    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:04:00.204650 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:04:00.204622    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:05:24.670789 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:24.670755    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:05:24.671391 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:24.671203    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="main" containerID="cri-o://c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d" gracePeriod=30
May 20 16:05:24.671391 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:24.671358    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="tokenizer" containerID="cri-o://ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80" gracePeriod=30
May 20 16:05:25.499053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.499017    2574 generic.go:358] "Generic (PLEG): container finished" podID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerID="c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d" exitCode=0
May 20 16:05:25.499242 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.499107    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerDied","Data":"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"}
May 20 16:05:25.827427 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.827405    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:05:25.896481 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896452    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896634 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896494    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896634 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896532    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896634 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896593    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896634 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896619    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896850 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896741    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cwp8\" (UniqueName: \"kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8\") pod \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\" (UID: \"ef890ab7-b1bd-4c30-bb08-01264d25b4df\") "
May 20 16:05:25.896850 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896756    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:25.896954 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896826    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:25.897010 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896962    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:25.897010 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.896995    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:25.897105 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.897015    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:25.897220 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.897199    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:25.898713 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.898685    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8" (OuterVolumeSpecName: "kube-api-access-5cwp8") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "kube-api-access-5cwp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:25.898713 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.898702    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef890ab7-b1bd-4c30-bb08-01264d25b4df" (UID: "ef890ab7-b1bd-4c30-bb08-01264d25b4df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:25.998395 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.998364    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:25.998395 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.998391    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:25.998395 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.998401    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwp8\" (UniqueName: \"kubernetes.io/projected/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kube-api-access-5cwp8\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:25.998609 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:25.998410    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef890ab7-b1bd-4c30-bb08-01264d25b4df-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:05:26.503366 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.503330    2574 generic.go:358] "Generic (PLEG): container finished" podID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerID="ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80" exitCode=0
May 20 16:05:26.503526 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.503376    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerDied","Data":"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"}
May 20 16:05:26.503526 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.503414    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6" event={"ID":"ef890ab7-b1bd-4c30-bb08-01264d25b4df","Type":"ContainerDied","Data":"52757ea9f3ce78ef27ca239f39163e1a510f3ce7894f0375fb6f7961ca7d1b75"}
May 20 16:05:26.503526 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.503415    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"
May 20 16:05:26.503526 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.503428    2574 scope.go:117] "RemoveContainer" containerID="ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"
May 20 16:05:26.511713 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.511695    2574 scope.go:117] "RemoveContainer" containerID="c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"
May 20 16:05:26.519543 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.519526    2574 scope.go:117] "RemoveContainer" containerID="c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"
May 20 16:05:26.524355 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.524327    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:05:26.527974 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.527957    2574 scope.go:117] "RemoveContainer" containerID="ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"
May 20 16:05:26.528275 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:05:26.528254    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80\": container with ID starting with ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80 not found: ID does not exist" containerID="ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"
May 20 16:05:26.528360 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528282    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80"} err="failed to get container status \"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80\": rpc error: code = NotFound desc = could not find container \"ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80\": container with ID starting with ddaa84e736690ff5475647bff895ca2eb00907a34d7a5e331098de911c2bbc80 not found: ID does not exist"
May 20 16:05:26.528360 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528299    2574 scope.go:117] "RemoveContainer" containerID="c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"
May 20 16:05:26.528556 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:05:26.528538    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d\": container with ID starting with c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d not found: ID does not exist" containerID="c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"
May 20 16:05:26.528612 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528560    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d"} err="failed to get container status \"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d\": rpc error: code = NotFound desc = could not find container \"c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d\": container with ID starting with c2f6b70237bf61a83a396c772283b4617e0f474e490fc18b20700ddc0d8e190d not found: ID does not exist"
May 20 16:05:26.528612 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528575    2574 scope.go:117] "RemoveContainer" containerID="c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"
May 20 16:05:26.528731 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528712    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6fd6c6d4mqcz6"]
May 20 16:05:26.528833 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:05:26.528806    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123\": container with ID starting with c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123 not found: ID does not exist" containerID="c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"
May 20 16:05:26.528888 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:26.528841    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123"} err="failed to get container status \"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123\": rpc error: code = NotFound desc = could not find container \"c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123\": container with ID starting with c50215f0ee08a4904e41fc3979cf3bf8f0120f4e4da67bceaa8ce981cf030123 not found: ID does not exist"
May 20 16:05:27.865183 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:27.864666    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" path="/var/lib/kubelet/pods/ef890ab7-b1bd-4c30-bb08-01264d25b4df/volumes"
May 20 16:05:35.041298 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041265    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041550    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="main"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041562    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="main"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041571    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="storage-initializer"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041577    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="storage-initializer"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041584    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="tokenizer"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041589    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="tokenizer"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041656    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="tokenizer"
May 20 16:05:35.041729 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.041666    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="ef890ab7-b1bd-4c30-bb08-01264d25b4df" containerName="main"
May 20 16:05:35.046581 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.046560    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.050598 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.050577    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-klv5d\""
May 20 16:05:35.050717 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.050600    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 20 16:05:35.055925 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.055904    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:05:35.174606 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.174774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.174774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.174774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.174883 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsxk\" (UniqueName: \"kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.174883 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.174811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275396 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275368    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275559 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275422    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275559 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275479    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275559 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275506    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsxk\" (UniqueName: \"kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275559 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275534    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275806 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275781    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275890 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275848    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275890 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275866    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.275999 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.275945    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.276139 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.276123    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.277940 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.277921    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.284231 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.284211    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsxk\" (UniqueName: \"kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk\") pod \"router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.355153 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.355127    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:35.485465 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.485427    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:05:35.488781 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:05:35.488754    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de39b0e_773b_4885_85da_7695dc9b69eb.slice/crio-5a5cc6cde60009ca5c3d1212d2b7a610f77ac8261dcedfc67140703ecdf8f34c WatchSource:0}: Error finding container 5a5cc6cde60009ca5c3d1212d2b7a610f77ac8261dcedfc67140703ecdf8f34c: Status 404 returned error can't find the container with id 5a5cc6cde60009ca5c3d1212d2b7a610f77ac8261dcedfc67140703ecdf8f34c
May 20 16:05:35.490928 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.490914    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:05:35.530476 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:35.530447    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerStarted","Data":"5a5cc6cde60009ca5c3d1212d2b7a610f77ac8261dcedfc67140703ecdf8f34c"}
May 20 16:05:36.534666 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:36.534581    2574 generic.go:358] "Generic (PLEG): container finished" podID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerID="52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6" exitCode=0
May 20 16:05:36.535024 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:36.534667    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerDied","Data":"52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6"}
May 20 16:05:37.540810 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:37.540777    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerStarted","Data":"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"}
May 20 16:05:37.540810 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:37.540812    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerStarted","Data":"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"}
May 20 16:05:37.541337 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:37.540886    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:37.563613 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:37.563566    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" podStartSLOduration=2.563552476 podStartE2EDuration="2.563552476s" podCreationTimestamp="2026-05-20 16:05:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:05:37.561214699 +0000 UTC m=+2036.301965378" watchObservedRunningTime="2026-05-20 16:05:37.563552476 +0000 UTC m=+2036.304303154"
May 20 16:05:45.356415 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:45.356371    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:45.356415 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:45.356418    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:45.358911 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:45.358887    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:05:45.573834 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:05:45.573804    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:06:06.576431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:06:06.576353    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:07:31.165881 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:31.165835    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:07:31.166441 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:31.166218    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="main" containerID="cri-o://aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b" gracePeriod=30
May 20 16:07:31.166441 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:31.166279    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="tokenizer" containerID="cri-o://7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab" gracePeriod=30
May 20 16:07:31.863547 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:31.863510    2574 generic.go:358] "Generic (PLEG): container finished" podID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerID="aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b" exitCode=0
May 20 16:07:31.863712 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:31.863582    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerDied","Data":"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"}
May 20 16:07:32.324107 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.324055    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:07:32.436518 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436438    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436518 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436504    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436790 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436560    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436790 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436596    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjsxk\" (UniqueName: \"kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436790 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436637    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436790 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436675    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache\") pod \"5de39b0e-773b-4885-85da-7695dc9b69eb\" (UID: \"5de39b0e-773b-4885-85da-7695dc9b69eb\") "
May 20 16:07:32.436943 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.436830    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:07:32.437233 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.437202    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:07:32.437519 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.437475    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:07:32.437809 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.437774    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:07:32.437809 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.437786    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.443647 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.443619    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk" (OuterVolumeSpecName: "kube-api-access-zjsxk") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "kube-api-access-zjsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:07:32.443817 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.443794    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5de39b0e-773b-4885-85da-7695dc9b69eb" (UID: "5de39b0e-773b-4885-85da-7695dc9b69eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:07:32.539047 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.539010    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.539047 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.539044    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjsxk\" (UniqueName: \"kubernetes.io/projected/5de39b0e-773b-4885-85da-7695dc9b69eb-kube-api-access-zjsxk\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.539254 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.539060    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de39b0e-773b-4885-85da-7695dc9b69eb-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.539254 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.539072    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.539254 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.539111    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5de39b0e-773b-4885-85da-7695dc9b69eb-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:07:32.867914 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.867877    2574 generic.go:358] "Generic (PLEG): container finished" podID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerID="7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab" exitCode=0
May 20 16:07:32.868096 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.867946    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerDied","Data":"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"}
May 20 16:07:32.868096 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.867968    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z" event={"ID":"5de39b0e-773b-4885-85da-7695dc9b69eb","Type":"ContainerDied","Data":"5a5cc6cde60009ca5c3d1212d2b7a610f77ac8261dcedfc67140703ecdf8f34c"}
May 20 16:07:32.868096 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.867971    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"
May 20 16:07:32.868096 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.867982    2574 scope.go:117] "RemoveContainer" containerID="7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"
May 20 16:07:32.876566 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.876548    2574 scope.go:117] "RemoveContainer" containerID="aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"
May 20 16:07:32.884974 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.884954    2574 scope.go:117] "RemoveContainer" containerID="52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6"
May 20 16:07:32.890924 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.890902    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:07:32.892510 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.892494    2574 scope.go:117] "RemoveContainer" containerID="7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"
May 20 16:07:32.892764 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:07:32.892747    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab\": container with ID starting with 7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab not found: ID does not exist" containerID="7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"
May 20 16:07:32.892814 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.892774    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab"} err="failed to get container status \"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab\": rpc error: code = NotFound desc = could not find container \"7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab\": container with ID starting with 7e2f1c54fa67d6b4bfc6ba78411306a92e006cb31aee2a3da6795dac7a3cc3ab not found: ID does not exist"
May 20 16:07:32.892814 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.892792    2574 scope.go:117] "RemoveContainer" containerID="aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"
May 20 16:07:32.893006 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:07:32.892993    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b\": container with ID starting with aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b not found: ID does not exist" containerID="aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"
May 20 16:07:32.893058 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.893008    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b"} err="failed to get container status \"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b\": rpc error: code = NotFound desc = could not find container \"aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b\": container with ID starting with aa9013a74417221ef743b79b5276990f1893261ee321c2391bcd0f464582093b not found: ID does not exist"
May 20 16:07:32.893058 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.893021    2574 scope.go:117] "RemoveContainer" containerID="52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6"
May 20 16:07:32.893255 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:07:32.893237    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6\": container with ID starting with 52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6 not found: ID does not exist" containerID="52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6"
May 20 16:07:32.893294 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.893262    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6"} err="failed to get container status \"52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6\": rpc error: code = NotFound desc = could not find container \"52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6\": container with ID starting with 52cc111ab512b8e21734eb0170d83b429c0dc8ab9cee44d7f37a9c37551586e6 not found: ID does not exist"
May 20 16:07:32.896112 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:32.896072    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-67c45b9c9c-npn7z"]
May 20 16:07:33.861734 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:33.861706    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" path="/var/lib/kubelet/pods/5de39b0e-773b-4885-85da-7695dc9b69eb/volumes"
May 20 16:07:41.800548 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800516    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800799    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="storage-initializer"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800810    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="storage-initializer"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800819    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="tokenizer"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800825    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="tokenizer"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800835    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="main"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800841    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="main"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800896    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="tokenizer"
May 20 16:07:41.800902 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.800904    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="5de39b0e-773b-4885-85da-7695dc9b69eb" containerName="main"
May 20 16:07:41.804230 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.804211    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.806761 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.806739    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-c6bk6\""
May 20 16:07:41.806761 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.806755    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 20 16:07:41.814897 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.814877    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:07:41.920133 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.920298 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.920298 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.920298 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.920454 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:41.920454 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:41.920366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pvt\" (UniqueName: \"kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021631 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021593    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021631 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021642    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021686    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021707    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pvt\" (UniqueName: \"kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021734    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.021950 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.021765    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.022544 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.022023    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.022544 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.022035    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.022544 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.022181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.022544 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.022300    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.024430 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.024410    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.030915 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.030893    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pvt\" (UniqueName: \"kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.112568 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.112537    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:42.267597 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.267574    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:07:42.268790 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:07:42.268764    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b18c13_9f03_4d95_a4ca_dc5df9458d3b.slice/crio-8c7e487b05407611ac1bae8e67b7f7ab550a045d41d9e2de7d18c88caca5bd1d WatchSource:0}: Error finding container 8c7e487b05407611ac1bae8e67b7f7ab550a045d41d9e2de7d18c88caca5bd1d: Status 404 returned error can't find the container with id 8c7e487b05407611ac1bae8e67b7f7ab550a045d41d9e2de7d18c88caca5bd1d
May 20 16:07:42.894779 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.894744    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerStarted","Data":"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"}
May 20 16:07:42.894779 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:42.894779    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerStarted","Data":"8c7e487b05407611ac1bae8e67b7f7ab550a045d41d9e2de7d18c88caca5bd1d"}
May 20 16:07:43.898745 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:43.898707    2574 generic.go:358] "Generic (PLEG): container finished" podID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerID="926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a" exitCode=0
May 20 16:07:43.899177 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:43.898793    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerDied","Data":"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"}
May 20 16:07:44.903614 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:44.903572    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerStarted","Data":"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"}
May 20 16:07:44.904002 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:44.903623    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerStarted","Data":"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"}
May 20 16:07:44.904002 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:44.903703    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:44.943470 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:44.943419    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" podStartSLOduration=3.94340415 podStartE2EDuration="3.94340415s" podCreationTimestamp="2026-05-20 16:07:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:07:44.940907601 +0000 UTC m=+2163.681658279" watchObservedRunningTime="2026-05-20 16:07:44.94340415 +0000 UTC m=+2163.684154828"
May 20 16:07:52.113457 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:52.113424    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:52.113868 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:52.113470    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:52.116062 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:52.116036    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:07:52.927991 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:07:52.927964    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:08:13.931282 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:08:13.931248    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:10:20.871013 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:20.870981    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:10:20.871535 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:20.871329    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="main" containerID="cri-o://78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3" gracePeriod=30
May 20 16:10:20.871535 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:20.871381    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="tokenizer" containerID="cri-o://86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da" gracePeriod=30
May 20 16:10:21.351930 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:21.351895    2574 generic.go:358] "Generic (PLEG): container finished" podID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerID="78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3" exitCode=0
May 20 16:10:21.352113 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:21.351979    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerDied","Data":"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"}
May 20 16:10:22.013369 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.013347    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:10:22.084313 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084283    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084479 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084325    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084479 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084360    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084479 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084385    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084479 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084417    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084479 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084477    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6pvt\" (UniqueName: \"kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt\") pod \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\" (UID: \"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b\") "
May 20 16:10:22.084742 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084637    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:22.084742 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084649    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:22.084742 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.084727    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:22.085060 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.085035    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:22.086351 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.086328    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:10:22.086491 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.086472    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt" (OuterVolumeSpecName: "kube-api-access-b6pvt") pod "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" (UID: "d4b18c13-9f03-4d95-a4ca-dc5df9458d3b"). InnerVolumeSpecName "kube-api-access-b6pvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:10:22.186015 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.185989    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.186015 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.186013    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.186208 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.186025    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.186208 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.186049    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.186208 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.186060    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.186208 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.186069    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pvt\" (UniqueName: \"kubernetes.io/projected/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b-kube-api-access-b6pvt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:10:22.356053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.355957    2574 generic.go:358] "Generic (PLEG): container finished" podID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerID="86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da" exitCode=0
May 20 16:10:22.356053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.355992    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerDied","Data":"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"}
May 20 16:10:22.356053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.356032    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25" event={"ID":"d4b18c13-9f03-4d95-a4ca-dc5df9458d3b","Type":"ContainerDied","Data":"8c7e487b05407611ac1bae8e67b7f7ab550a045d41d9e2de7d18c88caca5bd1d"}
May 20 16:10:22.356053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.356041    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"
May 20 16:10:22.356053 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.356049    2574 scope.go:117] "RemoveContainer" containerID="86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"
May 20 16:10:22.363909 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.363886    2574 scope.go:117] "RemoveContainer" containerID="78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"
May 20 16:10:22.370769 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.370753    2574 scope.go:117] "RemoveContainer" containerID="926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"
May 20 16:10:22.378688 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.378665    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:10:22.379173 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379155    2574 scope.go:117] "RemoveContainer" containerID="86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"
May 20 16:10:22.379424 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:10:22.379405    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da\": container with ID starting with 86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da not found: ID does not exist" containerID="86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"
May 20 16:10:22.379488 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379431    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da"} err="failed to get container status \"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da\": rpc error: code = NotFound desc = could not find container \"86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da\": container with ID starting with 86b7d2ea7f0c423ca54b44601480f2849aa2a0769b0604de1712f1c3f6ec70da not found: ID does not exist"
May 20 16:10:22.379488 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379447    2574 scope.go:117] "RemoveContainer" containerID="78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"
May 20 16:10:22.379627 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:10:22.379612    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3\": container with ID starting with 78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3 not found: ID does not exist" containerID="78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"
May 20 16:10:22.379675 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379630    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3"} err="failed to get container status \"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3\": rpc error: code = NotFound desc = could not find container \"78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3\": container with ID starting with 78e4362e6020570c750bb102198110ff20d26418438cfc9288ae14c48c988fd3 not found: ID does not exist"
May 20 16:10:22.379675 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379643    2574 scope.go:117] "RemoveContainer" containerID="926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"
May 20 16:10:22.379866 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:10:22.379843    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a\": container with ID starting with 926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a not found: ID does not exist" containerID="926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"
May 20 16:10:22.379923 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.379871    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a"} err="failed to get container status \"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a\": rpc error: code = NotFound desc = could not find container \"926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a\": container with ID starting with 926cbd868a61a175a01f7698190d5c07757e1e05277a7c51fcfa096bcece0c7a not found: ID does not exist"
May 20 16:10:22.381928 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:22.381900    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schebmz25"]
May 20 16:10:23.861206 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:10:23.861172    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" path="/var/lib/kubelet/pods/d4b18c13-9f03-4d95-a4ca-dc5df9458d3b/volumes"
May 20 16:12:58.132942 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.132908    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:12:58.133358 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133325    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="main"
May 20 16:12:58.133358 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133341    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="main"
May 20 16:12:58.133431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133362    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="tokenizer"
May 20 16:12:58.133431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133372    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="tokenizer"
May 20 16:12:58.133431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133393    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="storage-initializer"
May 20 16:12:58.133431 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133402    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="storage-initializer"
May 20 16:12:58.133555 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133471    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="tokenizer"
May 20 16:12:58.133555 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.133483    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4b18c13-9f03-4d95-a4ca-dc5df9458d3b" containerName="main"
May 20 16:12:58.136866 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.136846    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.139231 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.139205    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-qj86x\""
May 20 16:12:58.139346 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.139225    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 20 16:12:58.145668 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.145647    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:12:58.227917 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.227886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.228059 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.227927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4m6\" (UniqueName: \"kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.228059 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.227948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.228174 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.228050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.228174 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.228133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.228174 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.228167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329157 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329107    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4m6\" (UniqueName: \"kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329342 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329177    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329342 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329238    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329342 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329293    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329342 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329333    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329549 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329365    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329624 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329574    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329684 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329658    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329736 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329712    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.329773 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.329748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.331664 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.331647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.338988 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.338963    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4m6\" (UniqueName: \"kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.446720 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.446651    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:12:58.572739 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.572708    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:12:58.574686 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:12:58.574658    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfa185a_995d_42fc_b4fa_e7e22dff4657.slice/crio-47d0c0a85ee54fef262aa8f5ffd81eb55472c0d0388258ef899f9254e7600027 WatchSource:0}: Error finding container 47d0c0a85ee54fef262aa8f5ffd81eb55472c0d0388258ef899f9254e7600027: Status 404 returned error can't find the container with id 47d0c0a85ee54fef262aa8f5ffd81eb55472c0d0388258ef899f9254e7600027
May 20 16:12:58.582674 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.576963    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:12:58.790927 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.790839    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerStarted","Data":"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"}
May 20 16:12:58.790927 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:58.790879    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerStarted","Data":"47d0c0a85ee54fef262aa8f5ffd81eb55472c0d0388258ef899f9254e7600027"}
May 20 16:12:59.798065 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:59.797981    2574 generic.go:358] "Generic (PLEG): container finished" podID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerID="dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe" exitCode=0
May 20 16:12:59.798065 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:12:59.798024    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerDied","Data":"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"}
May 20 16:13:00.803029 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:00.802993    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerStarted","Data":"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"}
May 20 16:13:00.803029 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:00.803033    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerStarted","Data":"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"}
May 20 16:13:00.803475 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:00.803147    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:13:00.824889 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:00.824838    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" podStartSLOduration=2.824821938 podStartE2EDuration="2.824821938s" podCreationTimestamp="2026-05-20 16:12:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:13:00.823868933 +0000 UTC m=+2479.564619610" watchObservedRunningTime="2026-05-20 16:13:00.824821938 +0000 UTC m=+2479.565572618"
May 20 16:13:08.447308 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:08.447273    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:13:08.447718 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:08.447318    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:13:08.449812 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:08.449789    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:13:08.826417 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:08.826333    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:13:29.829242 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:13:29.829214    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:14:20.928186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:20.928128    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 16:14:20.928764 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:20.928539    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="main" containerID="cri-o://30cc6c2563b79eb6d8c3fb446df88a1880f2baf46ff477eb2a3cb0157812fa08" gracePeriod=30
May 20 16:14:20.928764 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:20.928630    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="tokenizer" containerID="cri-o://b2d64bf25e4ced2667bb0a74f426d30e7a20e159a980211fafbce9ef0b2f6aea" gracePeriod=30
May 20 16:14:22.026240 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.026209    2574 generic.go:358] "Generic (PLEG): container finished" podID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerID="b2d64bf25e4ced2667bb0a74f426d30e7a20e159a980211fafbce9ef0b2f6aea" exitCode=0
May 20 16:14:22.026240 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.026238    2574 generic.go:358] "Generic (PLEG): container finished" podID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerID="30cc6c2563b79eb6d8c3fb446df88a1880f2baf46ff477eb2a3cb0157812fa08" exitCode=0
May 20 16:14:22.026605 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.026278    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerDied","Data":"b2d64bf25e4ced2667bb0a74f426d30e7a20e159a980211fafbce9ef0b2f6aea"}
May 20 16:14:22.026605 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.026311    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerDied","Data":"30cc6c2563b79eb6d8c3fb446df88a1880f2baf46ff477eb2a3cb0157812fa08"}
May 20 16:14:22.072004 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.071984    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 16:14:22.107917 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.107889    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108061 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.107971    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108061 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108001    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw6jk\" (UniqueName: \"kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108061 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108043    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108247 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108097    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108247 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108158    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location\") pod \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\" (UID: \"b38b5523-4b86-4537-b31f-d3a6b68e5da1\") "
May 20 16:14:22.108339 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108250    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:22.108339 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108319    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:22.108460 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108440    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:22.108460 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108449    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:22.108574 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108467    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:22.108862 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.108840    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:22.110011 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.109989    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:14:22.110144 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.110118    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk" (OuterVolumeSpecName: "kube-api-access-qw6jk") pod "b38b5523-4b86-4537-b31f-d3a6b68e5da1" (UID: "b38b5523-4b86-4537-b31f-d3a6b68e5da1"). InnerVolumeSpecName "kube-api-access-qw6jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:14:22.209158 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.209072    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:22.209158 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.209112    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:22.209158 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.209122    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b38b5523-4b86-4537-b31f-d3a6b68e5da1-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:22.209158 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:22.209133    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6jk\" (UniqueName: \"kubernetes.io/projected/b38b5523-4b86-4537-b31f-d3a6b68e5da1-kube-api-access-qw6jk\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:14:23.030215 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.030179    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw" event={"ID":"b38b5523-4b86-4537-b31f-d3a6b68e5da1","Type":"ContainerDied","Data":"bca09ee98c9c1b41250e65b4de3e0027bcda2ab760c105e621afc8627595d6d7"}
May 20 16:14:23.030215 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.030200    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"
May 20 16:14:23.030734 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.030227    2574 scope.go:117] "RemoveContainer" containerID="b2d64bf25e4ced2667bb0a74f426d30e7a20e159a980211fafbce9ef0b2f6aea"
May 20 16:14:23.038242 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.038223    2574 scope.go:117] "RemoveContainer" containerID="30cc6c2563b79eb6d8c3fb446df88a1880f2baf46ff477eb2a3cb0157812fa08"
May 20 16:14:23.045051 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.045033    2574 scope.go:117] "RemoveContainer" containerID="2ecc8d47042144dbb18978828cf8da9a9907e96af5fa13efd84eebbbfa65c852"
May 20 16:14:23.054176 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.054156    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 16:14:23.065915 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.065890    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-764944jgsw"]
May 20 16:14:23.861246 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:23.861212    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" path="/var/lib/kubelet/pods/b38b5523-4b86-4537-b31f-d3a6b68e5da1/volumes"
May 20 16:14:53.428698 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428665    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428932    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="main"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428943    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="main"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428952    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="storage-initializer"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428958    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="storage-initializer"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428972    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="tokenizer"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.428978    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="tokenizer"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.429024    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="main"
May 20 16:14:53.429186 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.429031    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="b38b5523-4b86-4537-b31f-d3a6b68e5da1" containerName="tokenizer"
May 20 16:14:53.435373 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.435347    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.437984 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.437958    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-dc5fw\""
May 20 16:14:53.438851 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.438782    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 20 16:14:53.440594 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.440573    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:14:53.548006 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.547969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xrp\" (UniqueName: \"kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.548184 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.548023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.548184 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.548097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.548184 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.548128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.548311 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.548196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.548311 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.548241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.648946 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.648905    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649170 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.648961    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649170 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649000    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649170 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649043    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649170 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649079    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649170 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649145    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xrp\" (UniqueName: \"kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649443 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649389    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649443 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649412    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649534 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649450    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.649572 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.649561    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.651566 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.651550    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.656268 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.656252    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xrp\" (UniqueName: \"kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.745291 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.745209    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:53.870706 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:53.870679    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:14:53.873131 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:14:53.873100    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e8c8ea_90e8_4e73_aa9f_2e9fd8372d2e.slice/crio-78023ec1be46e5d0dfe316440fcc072ca03bbacd61e472af0d7ac71a826b92d1 WatchSource:0}: Error finding container 78023ec1be46e5d0dfe316440fcc072ca03bbacd61e472af0d7ac71a826b92d1: Status 404 returned error can't find the container with id 78023ec1be46e5d0dfe316440fcc072ca03bbacd61e472af0d7ac71a826b92d1
May 20 16:14:54.115414 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:54.115370    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerStarted","Data":"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"}
May 20 16:14:54.115414 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:54.115412    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerStarted","Data":"78023ec1be46e5d0dfe316440fcc072ca03bbacd61e472af0d7ac71a826b92d1"}
May 20 16:14:55.119330 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:55.119294    2574 generic.go:358] "Generic (PLEG): container finished" podID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerID="28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a" exitCode=0
May 20 16:14:55.119696 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:55.119343    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerDied","Data":"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"}
May 20 16:14:56.123498 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:56.123455    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerStarted","Data":"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"}
May 20 16:14:56.123498 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:56.123496    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerStarted","Data":"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"}
May 20 16:14:56.123933 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:56.123590    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:14:56.142546 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:14:56.142505    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" podStartSLOduration=3.142487417 podStartE2EDuration="3.142487417s" podCreationTimestamp="2026-05-20 16:14:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:14:56.140886855 +0000 UTC m=+2594.881637533" watchObservedRunningTime="2026-05-20 16:14:56.142487417 +0000 UTC m=+2594.883238097"
May 20 16:15:03.745444 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:03.745359    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:15:03.745444 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:03.745415    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:15:03.748043 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:03.748019    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:15:04.150764 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:04.150734    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:15:04.405631 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:04.405546    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:15:04.406004 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:04.405969    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="main" containerID="cri-o://bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5" gracePeriod=30
May 20 16:15:04.406169 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:04.406062    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="tokenizer" containerID="cri-o://70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8" gracePeriod=30
May 20 16:15:05.153761 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.153725    2574 generic.go:358] "Generic (PLEG): container finished" podID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerID="bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5" exitCode=0
May 20 16:15:05.154270 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.153802    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerDied","Data":"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"}
May 20 16:15:05.557822 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.557802    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:15:05.649757 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649727    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd4m6\" (UniqueName: \"kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.649927 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649787    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.649927 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649805    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.649927 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649915    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.650121 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649949    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.650121 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.649994    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache\") pod \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\" (UID: \"2dfa185a-995d-42fc-b4fa-e7e22dff4657\") "
May 20 16:15:05.650121 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.650005    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.650271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.650246    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:05.650347 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.650298    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.650347 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.650311    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.650602 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.650581    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.651874 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.651852    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:15:05.651945 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.651891    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6" (OuterVolumeSpecName: "kube-api-access-xd4m6") pod "2dfa185a-995d-42fc-b4fa-e7e22dff4657" (UID: "2dfa185a-995d-42fc-b4fa-e7e22dff4657"). InnerVolumeSpecName "kube-api-access-xd4m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:15:05.751271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.751197    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:05.751271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.751225    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4m6\" (UniqueName: \"kubernetes.io/projected/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kube-api-access-xd4m6\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:05.751271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.751239    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:05.751271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.751252    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:05.751271 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:05.751264    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dfa185a-995d-42fc-b4fa-e7e22dff4657-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:15:06.158306 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.158268    2574 generic.go:358] "Generic (PLEG): container finished" podID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerID="70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8" exitCode=0
May 20 16:15:06.158774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.158326    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerDied","Data":"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"}
May 20 16:15:06.158774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.158348    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"
May 20 16:15:06.158774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.158362    2574 scope.go:117] "RemoveContainer" containerID="70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"
May 20 16:15:06.158774 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.158351    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l" event={"ID":"2dfa185a-995d-42fc-b4fa-e7e22dff4657","Type":"ContainerDied","Data":"47d0c0a85ee54fef262aa8f5ffd81eb55472c0d0388258ef899f9254e7600027"}
May 20 16:15:06.165941 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.165907    2574 scope.go:117] "RemoveContainer" containerID="bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"
May 20 16:15:06.172446 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.172428    2574 scope.go:117] "RemoveContainer" containerID="dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"
May 20 16:15:06.174857 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.174832    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:15:06.179321 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.179298    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schetvq8l"]
May 20 16:15:06.182103 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.182048    2574 scope.go:117] "RemoveContainer" containerID="70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"
May 20 16:15:06.182564 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:15:06.182538    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8\": container with ID starting with 70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8 not found: ID does not exist" containerID="70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"
May 20 16:15:06.182627 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.182570    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8"} err="failed to get container status \"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8\": rpc error: code = NotFound desc = could not find container \"70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8\": container with ID starting with 70d025519758382da79d2da08ca026bd0c4dbb7d18fe18a068d2c939865ccbf8 not found: ID does not exist"
May 20 16:15:06.182627 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.182587    2574 scope.go:117] "RemoveContainer" containerID="bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"
May 20 16:15:06.182819 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:15:06.182805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5\": container with ID starting with bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5 not found: ID does not exist" containerID="bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"
May 20 16:15:06.182866 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.182822    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5"} err="failed to get container status \"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5\": rpc error: code = NotFound desc = could not find container \"bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5\": container with ID starting with bdaefeb5bc8b0064f6569c41f85559e493a1d4d0a8a1a7f94981a6105fa6f3c5 not found: ID does not exist"
May 20 16:15:06.182866 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.182838    2574 scope.go:117] "RemoveContainer" containerID="dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"
May 20 16:15:06.183051 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:15:06.183032    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe\": container with ID starting with dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe not found: ID does not exist" containerID="dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"
May 20 16:15:06.183135 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:06.183061    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe"} err="failed to get container status \"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe\": rpc error: code = NotFound desc = could not find container \"dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe\": container with ID starting with dca565281c0f70f5c271c18e8ad295d8c257f87ea3e122fecc56c62573e6b1fe not found: ID does not exist"
May 20 16:15:07.861329 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:07.861297    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" path="/var/lib/kubelet/pods/2dfa185a-995d-42fc-b4fa-e7e22dff4657/volumes"
May 20 16:15:25.155801 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:15:25.155772    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:16:54.701722 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:54.701628    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:16:54.702486 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:54.702007    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="main" containerID="cri-o://4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3" gracePeriod=30
May 20 16:16:54.702486 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:54.702313    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="tokenizer" containerID="cri-o://b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca" gracePeriod=30
May 20 16:16:55.155324 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:16:55.155298    2574 logging.go:55] [core] [Channel #1274 SubChannel #1275]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.37:9003", ServerName: "10.133.0.37:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.37:9003: connect: connection refused"
May 20 16:16:55.454757 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.454669    2574 generic.go:358] "Generic (PLEG): container finished" podID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerID="4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3" exitCode=0
May 20 16:16:55.454757 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.454728    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerDied","Data":"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"}
May 20 16:16:55.850547 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.850521    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:16:55.920988 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.920959    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921173 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921000    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xrp\" (UniqueName: \"kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921173 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921022    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921173 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921057    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921173 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921075    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921462 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921196    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location\") pod \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\" (UID: \"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e\") "
May 20 16:16:55.921462 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921313    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:16:55.921462 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921345    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:16:55.921565 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921483    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:16:55.921565 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921507    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-uds\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:55.921565 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921526    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:55.921895 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.921871    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:16:55.923204 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.923180    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:16:55.923273 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:55.923199    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp" (OuterVolumeSpecName: "kube-api-access-b9xrp") pod "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" (UID: "68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e"). InnerVolumeSpecName "kube-api-access-b9xrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:16:56.022365 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.022288    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xrp\" (UniqueName: \"kubernetes.io/projected/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kube-api-access-b9xrp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:56.022365 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.022316    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tokenizer-tmp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:56.022365 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.022326    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:56.022365 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.022335    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e-kserve-provision-location\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:16:56.155212 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.155175    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.37:9003\" within 1s: context deadline exceeded"
May 20 16:16:56.208899 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.208867    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2h7qd/must-gather-t4x92"]
May 20 16:16:56.209149 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209138    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="tokenizer"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209151    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="tokenizer"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209160    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="main"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209165    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="main"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209174    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="storage-initializer"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209179    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="storage-initializer"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209190    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="main"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209195    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="main"
May 20 16:16:56.209199 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209201    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="tokenizer"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209207    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="tokenizer"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209215    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="storage-initializer"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209220    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="storage-initializer"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209263    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="main"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209271    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerName="tokenizer"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209278    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="main"
May 20 16:16:56.209440 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.209286    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="2dfa185a-995d-42fc-b4fa-e7e22dff4657" containerName="tokenizer"
May 20 16:16:56.213258 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.213244    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.215754 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.215728    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2h7qd\"/\"openshift-service-ca.crt\""
May 20 16:16:56.215908 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.215887    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2h7qd\"/\"default-dockercfg-rxq4c\""
May 20 16:16:56.216027 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.215936    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2h7qd\"/\"kube-root-ca.crt\""
May 20 16:16:56.221065 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.221033    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2h7qd/must-gather-t4x92"]
May 20 16:16:56.324995 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.324900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xc2\" (UniqueName: \"kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.324995 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.324959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.425544 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.425507    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.425743 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.425580    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55xc2\" (UniqueName: \"kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.425913 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.425888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.433775 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.433752    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xc2\" (UniqueName: \"kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2\") pod \"must-gather-t4x92\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") " pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.459038 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.459008    2574 generic.go:358] "Generic (PLEG): container finished" podID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" containerID="b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca" exitCode=0
May 20 16:16:56.459181 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.459051    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerDied","Data":"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"}
May 20 16:16:56.459181 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.459076    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt" event={"ID":"68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e","Type":"ContainerDied","Data":"78023ec1be46e5d0dfe316440fcc072ca03bbacd61e472af0d7ac71a826b92d1"}
May 20 16:16:56.459181 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.459103    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"
May 20 16:16:56.459345 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.459107    2574 scope.go:117] "RemoveContainer" containerID="b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"
May 20 16:16:56.467530 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.467512    2574 scope.go:117] "RemoveContainer" containerID="4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"
May 20 16:16:56.474259 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.474238    2574 scope.go:117] "RemoveContainer" containerID="28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"
May 20 16:16:56.480495 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.480476    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:16:56.481638 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.481620    2574 scope.go:117] "RemoveContainer" containerID="b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"
May 20 16:16:56.481902 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:16:56.481869    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca\": container with ID starting with b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca not found: ID does not exist" containerID="b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"
May 20 16:16:56.481960 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.481915    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca"} err="failed to get container status \"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca\": rpc error: code = NotFound desc = could not find container \"b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca\": container with ID starting with b71b08ed2472408445ea53c8f953011235ca35b61448515105e211428610eaca not found: ID does not exist"
May 20 16:16:56.481960 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.481942    2574 scope.go:117] "RemoveContainer" containerID="4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"
May 20 16:16:56.482382 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:16:56.482290    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3\": container with ID starting with 4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3 not found: ID does not exist" containerID="4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"
May 20 16:16:56.482382 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.482338    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3"} err="failed to get container status \"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3\": rpc error: code = NotFound desc = could not find container \"4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3\": container with ID starting with 4b7a5d158171ac3ec4a95246f7f06f2eadc77553f966bff751574a1f13d935e3 not found: ID does not exist"
May 20 16:16:56.482382 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.482357    2574 scope.go:117] "RemoveContainer" containerID="28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"
May 20 16:16:56.482622 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:16:56.482597    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a\": container with ID starting with 28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a not found: ID does not exist" containerID="28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"
May 20 16:16:56.482682 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.482627    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a"} err="failed to get container status \"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a\": rpc error: code = NotFound desc = could not find container \"28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a\": container with ID starting with 28c56b1298f33e77e044c6ca47d3f5f01a64cd23da15b2be9793a7bdd5e1dc2a not found: ID does not exist"
May 20 16:16:56.483766 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.483745    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5dff66749trszt"]
May 20 16:16:56.523599 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.523573    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:16:56.642105 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:56.642059    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2h7qd/must-gather-t4x92"]
May 20 16:16:56.644836 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:16:56.644802    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89eacfe6_2ebf_4085_9c6d_30958ae74df0.slice/crio-b73edd89ff1e77a3fe0acfd6cf249103110ec8b89f56181c7d13094690aa1f9d WatchSource:0}: Error finding container b73edd89ff1e77a3fe0acfd6cf249103110ec8b89f56181c7d13094690aa1f9d: Status 404 returned error can't find the container with id b73edd89ff1e77a3fe0acfd6cf249103110ec8b89f56181c7d13094690aa1f9d
May 20 16:16:57.466078 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:57.466019    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2h7qd/must-gather-t4x92" event={"ID":"89eacfe6-2ebf-4085-9c6d-30958ae74df0","Type":"ContainerStarted","Data":"b73edd89ff1e77a3fe0acfd6cf249103110ec8b89f56181c7d13094690aa1f9d"}
May 20 16:16:57.862493 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:16:57.862461    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e" path="/var/lib/kubelet/pods/68e8c8ea-90e8-4e73-aa9f-2e9fd8372d2e/volumes"
May 20 16:17:03.487074 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:03.487033    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2h7qd/must-gather-t4x92" event={"ID":"89eacfe6-2ebf-4085-9c6d-30958ae74df0","Type":"ContainerStarted","Data":"2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b"}
May 20 16:17:03.487503 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:03.487100    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2h7qd/must-gather-t4x92" event={"ID":"89eacfe6-2ebf-4085-9c6d-30958ae74df0","Type":"ContainerStarted","Data":"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"}
May 20 16:17:03.502494 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:03.502446    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-2h7qd/must-gather-t4x92" podStartSLOduration=1.578370466 podStartE2EDuration="7.502431752s" podCreationTimestamp="2026-05-20 16:16:56 +0000 UTC" firstStartedPulling="2026-05-20 16:16:56.646469412 +0000 UTC m=+2715.387220068" lastFinishedPulling="2026-05-20 16:17:02.570530698 +0000 UTC m=+2721.311281354" observedRunningTime="2026-05-20 16:17:03.500308936 +0000 UTC m=+2722.241059616" watchObservedRunningTime="2026-05-20 16:17:03.502431752 +0000 UTC m=+2722.243182429"
May 20 16:17:11.631495 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:11.631464    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:12.690169 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:12.690134    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:13.699895 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:13.699851    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:14.679890 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:14.679859    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:15.686252 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:15.686213    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:16.686454 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:16.686416    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:17.682668 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:17.682630    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:18.676518 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:18.676485    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:19.663494 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:19.663456    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:20.639513 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:20.639486    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:21.659060 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:21.659030    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:22.710633 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:22.710608    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:23.760069 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:23.760042    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:24.821201 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:24.821176    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-7xtk4_177565c0-e6b6-4421-b395-986c4fc84d80/istio-proxy/0.log"
May 20 16:17:27.571596 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:27.571569    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-kkwsz_7b261504-b677-4bd0-b497-c29253d3ff3e/kuadrant-console-plugin/0.log"
May 20 16:17:28.562535 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:28.562447    2574 generic.go:358] "Generic (PLEG): container finished" podID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerID="1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538" exitCode=0
May 20 16:17:28.562755 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:28.562526    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2h7qd/must-gather-t4x92" event={"ID":"89eacfe6-2ebf-4085-9c6d-30958ae74df0","Type":"ContainerDied","Data":"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"}
May 20 16:17:28.562920 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:28.562904    2574 scope.go:117] "RemoveContainer" containerID="1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"
May 20 16:17:29.264059 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:29.264034    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2h7qd_must-gather-t4x92_89eacfe6-2ebf-4085-9c6d-30958ae74df0/gather/0.log"
May 20 16:17:34.757697 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.757646    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2h7qd/must-gather-t4x92"]
May 20 16:17:34.758330 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.757938    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-2h7qd/must-gather-t4x92" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="copy" containerID="cri-o://2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b" gracePeriod=2
May 20 16:17:34.759884 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.759856    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2h7qd/must-gather-t4x92"]
May 20 16:17:34.760019 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.759990    2574 status_manager.go:895] "Failed to get status for pod" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" pod="openshift-must-gather-2h7qd/must-gather-t4x92" err="pods \"must-gather-t4x92\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2h7qd\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object"
May 20 16:17:34.907259 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.907233    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g8b9j_f9696679-eb3e-4d6a-b899-a87da586bd90/global-pull-secret-syncer/0.log"
May 20 16:17:34.985671 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.985648    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2h7qd_must-gather-t4x92_89eacfe6-2ebf-4085-9c6d-30958ae74df0/copy/0.log"
May 20 16:17:34.986002 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.985987    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:17:34.987901 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:34.987880    2574 status_manager.go:895] "Failed to get status for pod" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" pod="openshift-must-gather-2h7qd/must-gather-t4x92" err="pods \"must-gather-t4x92\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2h7qd\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object"
May 20 16:17:35.044359 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.044280    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55xc2\" (UniqueName: \"kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2\") pod \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") "
May 20 16:17:35.044359 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.044345    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output\") pod \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\" (UID: \"89eacfe6-2ebf-4085-9c6d-30958ae74df0\") "
May 20 16:17:35.046553 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.046524    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2" (OuterVolumeSpecName: "kube-api-access-55xc2") pod "89eacfe6-2ebf-4085-9c6d-30958ae74df0" (UID: "89eacfe6-2ebf-4085-9c6d-30958ae74df0"). InnerVolumeSpecName "kube-api-access-55xc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:17:35.051286 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.051258    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "89eacfe6-2ebf-4085-9c6d-30958ae74df0" (UID: "89eacfe6-2ebf-4085-9c6d-30958ae74df0"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:35.058392 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.058372    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gdjgt_c2edd435-0b32-47a9-91ef-48736be5945b/konnectivity-agent/0.log"
May 20 16:17:35.122985 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.122963    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-222.ec2.internal_a5244faf83b4a2df37ec043783dad277/haproxy/0.log"
May 20 16:17:35.145075 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.145049    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89eacfe6-2ebf-4085-9c6d-30958ae74df0-must-gather-output\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:17:35.145075 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.145072    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xc2\" (UniqueName: \"kubernetes.io/projected/89eacfe6-2ebf-4085-9c6d-30958ae74df0-kube-api-access-55xc2\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\""
May 20 16:17:35.584171 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.584144    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2h7qd_must-gather-t4x92_89eacfe6-2ebf-4085-9c6d-30958ae74df0/copy/0.log"
May 20 16:17:35.584464 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.584442    2574 generic.go:358] "Generic (PLEG): container finished" podID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerID="2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b" exitCode=143
May 20 16:17:35.584533 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.584492    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2h7qd/must-gather-t4x92"
May 20 16:17:35.584599 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.584496    2574 scope.go:117] "RemoveContainer" containerID="2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b"
May 20 16:17:35.587667 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.587643    2574 status_manager.go:895] "Failed to get status for pod" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" pod="openshift-must-gather-2h7qd/must-gather-t4x92" err="pods \"must-gather-t4x92\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2h7qd\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object"
May 20 16:17:35.592302 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.592177    2574 scope.go:117] "RemoveContainer" containerID="1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"
May 20 16:17:35.594972 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.594946    2574 status_manager.go:895] "Failed to get status for pod" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" pod="openshift-must-gather-2h7qd/must-gather-t4x92" err="pods \"must-gather-t4x92\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2h7qd\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object"
May 20 16:17:35.604390 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.604373    2574 scope.go:117] "RemoveContainer" containerID="2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b"
May 20 16:17:35.604637 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:17:35.604621    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b\": container with ID starting with 2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b not found: ID does not exist" containerID="2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b"
May 20 16:17:35.604683 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.604645    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b"} err="failed to get container status \"2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b\": rpc error: code = NotFound desc = could not find container \"2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b\": container with ID starting with 2052d6f06635b98c2b16a81d8db7301729d3382c9a15cdf0cc3cbbdd95766c4b not found: ID does not exist"
May 20 16:17:35.604683 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.604663    2574 scope.go:117] "RemoveContainer" containerID="1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"
May 20 16:17:35.604891 ip-10-0-141-222 kubenswrapper[2574]: E0520 16:17:35.604874    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538\": container with ID starting with 1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538 not found: ID does not exist" containerID="1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"
May 20 16:17:35.604932 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.604898    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538"} err="failed to get container status \"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538\": rpc error: code = NotFound desc = could not find container \"1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538\": container with ID starting with 1c4b6c5545b11cc946a0430346d81f9b89fa91a5db0da3e246a0bff7aa3d8538 not found: ID does not exist"
May 20 16:17:35.861901 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:35.861821    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" path="/var/lib/kubelet/pods/89eacfe6-2ebf-4085-9c6d-30958ae74df0/volumes"
May 20 16:17:39.690428 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:39.690400    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-kkwsz_7b261504-b677-4bd0-b497-c29253d3ff3e/kuadrant-console-plugin/0.log"
May 20 16:17:41.195488 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:41.195458    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cm7ht_79c9f96f-9d52-4ff5-bec1-603d7211aa3f/node-exporter/0.log"
May 20 16:17:41.220155 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:41.220132    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cm7ht_79c9f96f-9d52-4ff5-bec1-603d7211aa3f/kube-rbac-proxy/0.log"
May 20 16:17:41.247381 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:41.247360    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cm7ht_79c9f96f-9d52-4ff5-bec1-603d7211aa3f/init-textfile/0.log"
May 20 16:17:44.279788 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.279752    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"]
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280021    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="gather"
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280032    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="gather"
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280041    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="copy"
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280046    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="copy"
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280109    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="copy"
May 20 16:17:44.280182 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.280119    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="89eacfe6-2ebf-4085-9c6d-30958ae74df0" containerName="gather"
May 20 16:17:44.282129 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.282115    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.284443 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.284419    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m228h\"/\"kube-root-ca.crt\""
May 20 16:17:44.284579 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.284442    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m228h\"/\"openshift-service-ca.crt\""
May 20 16:17:44.284579 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.284542    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m228h\"/\"default-dockercfg-br4zm\""
May 20 16:17:44.291183 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.291163    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"]
May 20 16:17:44.313739 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.313714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-podres\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.313842 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.313747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-proc\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.313842 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.313770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fc5\" (UniqueName: \"kubernetes.io/projected/9009b938-9e72-4f26-a67e-d38922f67aeb-kube-api-access-f2fc5\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.313842 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.313815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-sys\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.313944 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.313880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-lib-modules\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414623 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414587    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-lib-modules\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414773 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414640    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-podres\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414773 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414746    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-podres\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414773 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414749    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-proc\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414877 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-proc\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414877 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-lib-modules\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414877 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414793    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fc5\" (UniqueName: \"kubernetes.io/projected/9009b938-9e72-4f26-a67e-d38922f67aeb-kube-api-access-f2fc5\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.414877 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414864    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-sys\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.415002 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.414955    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9009b938-9e72-4f26-a67e-d38922f67aeb-sys\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.421712 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.421686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fc5\" (UniqueName: \"kubernetes.io/projected/9009b938-9e72-4f26-a67e-d38922f67aeb-kube-api-access-f2fc5\") pod \"perf-node-gather-daemonset-zcwzs\" (UID: \"9009b938-9e72-4f26-a67e-d38922f67aeb\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.592689 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.592658    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:44.715031 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:44.715007    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"]
May 20 16:17:44.717461 ip-10-0-141-222 kubenswrapper[2574]: W0520 16:17:44.717434    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9009b938_9e72_4f26_a67e_d38922f67aeb.slice/crio-24d08362fcbe97eb6dde2729773dd3abb3980a0ea69717509fd8fb733d8958e0 WatchSource:0}: Error finding container 24d08362fcbe97eb6dde2729773dd3abb3980a0ea69717509fd8fb733d8958e0: Status 404 returned error can't find the container with id 24d08362fcbe97eb6dde2729773dd3abb3980a0ea69717509fd8fb733d8958e0
May 20 16:17:45.224680 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.224583    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22rt8_29045f79-cc96-419a-8c83-dc73b9bd3bf0/dns/0.log"
May 20 16:17:45.243919 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.243892    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22rt8_29045f79-cc96-419a-8c83-dc73b9bd3bf0/kube-rbac-proxy/0.log"
May 20 16:17:45.351571 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.351542    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-747vx_d325a410-f630-445c-a538-feddddf807d2/dns-node-resolver/0.log"
May 20 16:17:45.617827 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.617790    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs" event={"ID":"9009b938-9e72-4f26-a67e-d38922f67aeb","Type":"ContainerStarted","Data":"d800aa3c27324c52825784f67d47242c1d2c6066172f6f95d25a40f17bbc016a"}
May 20 16:17:45.617827 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.617831    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs" event={"ID":"9009b938-9e72-4f26-a67e-d38922f67aeb","Type":"ContainerStarted","Data":"24d08362fcbe97eb6dde2729773dd3abb3980a0ea69717509fd8fb733d8958e0"}
May 20 16:17:45.618051 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.617906    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:45.633868 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.633825    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs" podStartSLOduration=1.633811024 podStartE2EDuration="1.633811024s" podCreationTimestamp="2026-05-20 16:17:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:17:45.631438121 +0000 UTC m=+2764.372188809" watchObservedRunningTime="2026-05-20 16:17:45.633811024 +0000 UTC m=+2764.374561702"
May 20 16:17:45.840307 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.840273    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-664d49788c-l46lh_bfd4294a-3da7-4dec-b02a-d7e53992e714/registry/0.log"
May 20 16:17:45.880179 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:45.880106    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cjl8v_f1736103-41b1-47b5-beac-d9442dd46cc0/node-ca/0.log"
May 20 16:17:47.277137 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:47.277107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zcsn_e5078f38-1131-44ce-8880-29fe9ccfd946/serve-healthcheck-canary/0.log"
May 20 16:17:47.770928 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:47.770902    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6xcj5_7f86fa47-b321-4b0b-940d-5160e189d8e8/kube-rbac-proxy/0.log"
May 20 16:17:47.790493 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:47.790473    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6xcj5_7f86fa47-b321-4b0b-940d-5160e189d8e8/exporter/0.log"
May 20 16:17:47.811451 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:47.811427    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6xcj5_7f86fa47-b321-4b0b-940d-5160e189d8e8/extractor/0.log"
May 20 16:17:50.410167 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:50.410135    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cf55c5997-2xlk9_d1a71f86-c7ec-4feb-b398-cd925eb5bc6d/manager/0.log"
May 20 16:17:51.246333 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:51.246302    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-279fk_83499bdf-28f6-4ed9-92da-484c629caa50/manager/0.log"
May 20 16:17:51.264098 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:51.264068    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jpvfk_5430d97e-96a6-4d1c-a1ea-d2bce943a5f9/s3-init/0.log"
May 20 16:17:51.629542 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:51.629510    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-zcwzs"
May 20 16:17:55.763831 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:55.763803    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-9drjl_bb60b51c-622c-4185-a4fa-8b894b7b1de3/migrator/0.log"
May 20 16:17:55.786165 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:55.786097    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-9drjl_bb60b51c-622c-4185-a4fa-8b894b7b1de3/graceful-termination/0.log"
May 20 16:17:57.311964 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.311883    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/kube-multus-additional-cni-plugins/0.log"
May 20 16:17:57.333254 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.333225    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/egress-router-binary-copy/0.log"
May 20 16:17:57.354641 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.354621    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/cni-plugins/0.log"
May 20 16:17:57.372855 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.372840    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/bond-cni-plugin/0.log"
May 20 16:17:57.392118 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.392095    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/routeoverride-cni/0.log"
May 20 16:17:57.409432 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.409413    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/whereabouts-cni-bincopy/0.log"
May 20 16:17:57.427315 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.427296    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ct8vp_43619788-ee3e-4703-a24c-29887c78e05f/whereabouts-cni/0.log"
May 20 16:17:57.602622 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.602597    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2rvk_e4d43c9f-a1da-4bb9-afe8-ad383db25c79/kube-multus/0.log"
May 20 16:17:57.705012 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.704978    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qmkpv_9e4f292c-23eb-45d1-be14-7e368f49ff38/network-metrics-daemon/0.log"
May 20 16:17:57.723569 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:57.723546    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qmkpv_9e4f292c-23eb-45d1-be14-7e368f49ff38/kube-rbac-proxy/0.log"
May 20 16:17:59.264853 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.264827    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/ovn-controller/0.log"
May 20 16:17:59.295032 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.295004    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/ovn-acl-logging/0.log"
May 20 16:17:59.310194 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.310173    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/kube-rbac-proxy-node/0.log"
May 20 16:17:59.328918 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.328900    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/kube-rbac-proxy-ovn-metrics/0.log"
May 20 16:17:59.347296 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.347273    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/northd/0.log"
May 20 16:17:59.365245 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.365228    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/nbdb/0.log"
May 20 16:17:59.385207 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.385191    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/sbdb/0.log"
May 20 16:17:59.481873 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:17:59.481842    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjrxd_4c34c181-6934-4c86-bf7d-71221abd751c/ovnkube-controller/0.log"
May 20 16:18:00.563684 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:18:00.563604    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dz424_64cd1e6e-498b-43df-9ebc-cdc2771a6ad2/network-check-target-container/0.log"
May 20 16:18:01.556759 ip-10-0-141-222 kubenswrapper[2574]: I0520 16:18:01.556729    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2gv4s_e4fd9469-a348-412a-abf5-c8ef100edb5e/iptables-alerter/0.log"