Jun 02 15:46:54.197355 ip-10-0-135-190 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:46:54.639564 ip-10-0-135-190 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:54.639564 ip-10-0-135-190 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:46:54.639564 ip-10-0-135-190 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:54.639564 ip-10-0-135-190 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:46:54.639564 ip-10-0-135-190 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:54.641567 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.641482    2568 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:46:54.644422 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644408    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:54.644422 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644422    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644426    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644430    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644433    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644435    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644438    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644441    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644443    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644446    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644448    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644451    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644454    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644456    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644459    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644462    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644464    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644466    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644469    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644472    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644474    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:54.644484 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644477    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644479    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644482    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644484    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644487    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644490    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644496    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644499    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644501    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644504    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644506    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644509    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644511    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644514    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644516    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644519    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644521    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644523    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644526    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644528    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:54.644953 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644531    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644533    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644535    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644538    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644541    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644543    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644545    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644548    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644550    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644553    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644556    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644558    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644562    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644564    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644567    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644570    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644573    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644575    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644578    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:54.645458 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644580    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644583    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644586    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644590    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644594    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644598    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644602    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644606    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644610    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644613    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644616    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644618    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644621    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644625    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644627    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644631    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644633    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644636    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644638    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:54.645961 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644641    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644644    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644646    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644650    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644654    2568 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644656    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.644660    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645029    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645035    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645038    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645041    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645044    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645066    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645069    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645072    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645075    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645077    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645080    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645082    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:54.646451 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645085    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645087    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645090    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645092    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645095    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645097    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645100    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645102    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645105    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645108    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645111    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645113    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645116    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645118    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645121    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645123    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645125    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645128    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645130    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645133    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:54.646916 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645136    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645139    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645141    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645143    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645146    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645149    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645151    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645154    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645156    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645159    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645161    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645164    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645166    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645169    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645171    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645174    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645177    2568 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645180    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645183    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:54.647416 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645185    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645188    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645191    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645194    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645197    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645200    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645202    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645205    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645207    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645210    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645212    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645215    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645218    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645221    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645224    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645227    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645229    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645232    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645234    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645237    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:54.647876 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645239    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645242    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645244    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645247    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645249    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645252    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645254    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645256    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645259    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645261    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645264    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645267    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645270    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645272    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.645275    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646755    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646767    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646774    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646779    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646783    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646786    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:46:54.648380 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646791    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646796    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646799    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646803    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646807    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646810    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646813    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646816    2568 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646819    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646822    2568 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646825    2568 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646828    2568 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646831    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646835    2568 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646838    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646841    2568 flags.go:64] FLAG: --config-dir=""
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646844    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646847    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646851    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646854    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646857    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646860    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646863    2568 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646866    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:46:54.648885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646869    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646872    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646878    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646883    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646887    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646890    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646892    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646895    2568 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646898    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646903    2568 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646906    2568 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646909    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646912    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646915    2568 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646919    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646922    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646925    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646928    2568 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646931    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646934    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646937    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646940    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646943    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646946    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646949    2568 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:46:54.649491 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646958    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646961    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646964    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646967    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646969    2568 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646973    2568 flags.go:64] FLAG: --help="false"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646976    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-135-190.ec2.internal"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646979    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646982    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646984    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646988    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646991    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646994    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.646997    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647000    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647003    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647006    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647009    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647012    2568 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647016    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647018    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647022    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647024    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647027    2568 flags.go:64] FLAG: --lock-file=""
Jun 02 15:46:54.650115 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647030    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647033    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647036    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647041    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647044    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647057    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647060    2568 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647063    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647066    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647069    2568 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647072    2568 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647076    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647079    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647083    2568 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647086    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647089    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647092    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647095    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647098    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647101    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647104    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647111    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647114    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647117    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:46:54.650738 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647120    2568 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647123    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647129    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647132    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647135    2568 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647138    2568 flags.go:64] FLAG: --port="10250"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647142    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647144    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027ba2dd2928ce9a9"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647148    2568 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647150    2568 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647153    2568 flags.go:64] FLAG: --register-node="true"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647156    2568 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647159    2568 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647162    2568 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647165    2568 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647168    2568 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647171    2568 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647174    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647177    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647180    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647183    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647186    2568 flags.go:64] FLAG: --runonce="false"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647189    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647191    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647194    2568 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:46:54.651341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647197    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647200    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647203    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647206    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647209    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647213    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647216    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647218    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647221    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647224    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647227    2568 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647230    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647236    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647238    2568 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647241    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647245    2568 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647248    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647251    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647254    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647257    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647260    2568 flags.go:64] FLAG: --v="2"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647264    2568 flags.go:64] FLAG: --version="false"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647268    2568 flags.go:64] FLAG: --vmodule=""
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647272    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647275    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:46:54.651931 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647367    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647370    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647373    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647376    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647379    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647382    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647385    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647388    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647390    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647393    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647395    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647402    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647405    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647408    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647411    2568 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647414    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647416    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647419    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647422    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647425    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:54.652574 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647427    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647430    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647433    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647435    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647438    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647440    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647443    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647446    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647448    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647451    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647453    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647456    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647459    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647461    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647464    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647466    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647469    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647472    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647474    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:54.653089 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647477    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647480    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647482    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647485    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647489    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647491    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647494    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647497    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647499    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647502    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647505    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647507    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647510    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647512    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647515    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647517    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647520    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647523    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647525    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647528    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:54.653628 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647530    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647533    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647535    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647538    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647540    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647543    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647545    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647548    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647551    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647553    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647555    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647558    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647560    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647563    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647566    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647570    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647574    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647577    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647581    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647583    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:54.654126 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647587    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647590    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647593    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647595    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647598    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647602    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.647606    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.647611    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.653645    2568 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.653660    2568 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653709    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653714    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653717    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653720    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653723    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:54.654624 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653726    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653728    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653731    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653734    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653737    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653739    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653741    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653745    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653750    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653753    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653756    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653759    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653762    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653765    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653768    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653772    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653775    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653779    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653781    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:54.655009 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653784    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653787    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653789    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653792    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653795    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653798    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653801    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653804    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653806    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653809    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653812    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653814    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653817    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653819    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653822    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653825    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653827    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653830    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653833    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653836    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653838    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:54.655483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653841    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653843    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653846    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653848    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653851    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653854    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653857    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653859    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653862    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653865    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653868    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653871    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653873    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653876    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653879    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653881    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653884    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653886    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653889    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:54.656003 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653892    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653895    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653897    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653900    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653902    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653905    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653907    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653910    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653912    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653915    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653917    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653920    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653922    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653924    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653927    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653929    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653932    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653935    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653938    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653941    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653943    2568 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:54.656483 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.653946    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.653951    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654062    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654068    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654071    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654074    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654076    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654079    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654082    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654085    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654088    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654090    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654093    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654096    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654099    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654101    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:54.656998 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654103    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654106    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654109    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654111    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654113    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654116    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654118    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654121    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654123    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654126    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654128    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654131    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654134    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654136    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654139    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654142    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654144    2568 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654147    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654150    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:54.657461 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654152    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654155    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654158    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654160    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654162    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654165    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654167    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654170    2568 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654173    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654175    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654178    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654180    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654183    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654185    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654188    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654190    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654194    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654198    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654202    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:54.657925 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654205    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654208    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654210    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654213    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654216    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654219    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654221    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654224    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654227    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654229    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654232    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654234    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654237    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654239    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654242    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654245    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654247    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654250    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654252    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654255    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:54.658386 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654257    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654260    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654262    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654265    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654267    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654270    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654272    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654275    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654277    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654280    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654282    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654285    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654287    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:54.654290    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.654294    2568 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:54.658888 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.655068    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:46:54.660590 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.660576    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:46:54.661520 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.661509    2568 server.go:1019] "Starting client certificate rotation"
Jun 02 15:46:54.661638 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.661612    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:54.661698 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.661665    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:54.687875 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.687855    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:54.693147 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.693127    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:54.712775 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.712755    2568 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:46:54.719095 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.719080    2568 log.go:25] "Validated CRI v1 image API"
Jun 02 15:46:54.719226 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.719210    2568 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:54.720280 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.720259    2568 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:46:54.722713 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.722687    2568 fs.go:135] Filesystem UUIDs: map[4161cc1a-ffc3-4386-9800-d34b4834c114:/dev/nvme0n1p4 66e235a8-9cb5-4c1d-bc33-8795ab2863eb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jun 02 15:46:54.722782 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.722712    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:46:54.727025 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.726920    2568 manager.go:217] Machine: {Timestamp:2026-06-02 15:46:54.726419881 +0000 UTC m=+0.410967807 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100407 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2518bc1e9ef65a7ff7566569ac64b4 SystemUUID:ec2518bc-1e9e-f65a-7ff7-566569ac64b4 BootID:07bd8ec3-3cd0-4497-afa6-5caa5cf34bbb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:74:43:c5:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:74:43:c5:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:9f:57:b6:e7:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:46:54.727121 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.727063    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:46:54.727162 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.727149    2568 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:46:54.730220 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.730190    2568 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:46:54.730394 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.730223    2568 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:46:54.730470 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.730409    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:46:54.730470 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.730424    2568 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:46:54.730470 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.730440    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:54.731193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.731181    2568 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:54.732914 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.732900    2568 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:54.733044 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.733033    2568 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:46:54.735321 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.735309    2568 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:46:54.735386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.735327    2568 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:46:54.735386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.735344    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:46:54.735386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.735357    2568 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:46:54.735386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.735369    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:46:54.736609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.736595    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:54.736693 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.736618    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:54.740433 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.740405    2568 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:46:54.742403 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.742386    2568 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:46:54.744146 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744135    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744151    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744158    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744163    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744172    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744178    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744187    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:46:54.744193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744193    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:46:54.744376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744201    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:46:54.744376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744207    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:46:54.744376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744216    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:46:54.744376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.744221    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:46:54.745033 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.745024    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:46:54.745087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.745035    2568 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:46:54.748575 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.748562    2568 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:46:54.748636 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.748597    2568 server.go:1295] "Started kubelet"
Jun 02 15:46:54.748721 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.748677    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:46:54.748771 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.748678    2568 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:46:54.748771 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.748764    2568 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:46:54.749416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.749395    2568 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:46:54.749416 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.749393    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:46:54.749557 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.749517    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:46:54.749601 ip-10-0-135-190 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:46:54.749859 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.749789    2568 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:46:54.751581 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.751567    2568 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:46:54.754475 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.753645    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-190.ec2.internal.18b54e346bca206c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-190.ec2.internal,UID:ip-10-0-135-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-190.ec2.internal,},FirstTimestamp:2026-06-02 15:46:54.748573804 +0000 UTC m=+0.433121730,LastTimestamp:2026-06-02 15:46:54.748573804 +0000 UTC m=+0.433121730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-190.ec2.internal,}"
Jun 02 15:46:54.756228 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.756210    2568 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:46:54.756847 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.756831    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:54.757334 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757323    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:46:54.757500 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757481    2568 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:46:54.757500 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757500    2568 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:46:54.757618 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757523    2568 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:46:54.757618 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757595    2568 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:46:54.757618 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757603    2568 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:46:54.757939 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757913    2568 factory.go:55] Registering systemd factory
Jun 02 15:46:54.757998 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.757966    2568 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:46:54.758211 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758192    2568 factory.go:153] Registering CRI-O factory
Jun 02 15:46:54.758211 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758212    2568 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:46:54.758349 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.758246    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:54.758349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758267    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:46:54.758349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758303    2568 factory.go:103] Registering Raw factory
Jun 02 15:46:54.758349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758329    2568 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:46:54.758957 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.758941    2568 manager.go:319] Starting recovery of all containers
Jun 02 15:46:54.770017 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.769851    2568 manager.go:324] Recovery completed
Jun 02 15:46:54.772479 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.772449    2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jun 02 15:46:54.774132 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.774111    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 02 15:46:54.774225 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.774136    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 02 15:46:54.775572 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.775558    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.777299 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.777270    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.777299 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.777300    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.777451 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.777310    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:54.778030 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.778014    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:46:54.778030 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.778025    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:46:54.778183 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.778041    2568 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:54.780325 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.780312    2568 policy_none.go:49] "None policy: Start"
Jun 02 15:46:54.780385 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.780329    2568 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:46:54.780385 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.780340    2568 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:46:54.786117 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.786099    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qt6cw"
Jun 02 15:46:54.786615 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.786547    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-190.ec2.internal.18b54e346d8040bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-190.ec2.internal,UID:ip-10-0-135-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-190.ec2.internal,},FirstTimestamp:2026-06-02 15:46:54.777286843 +0000 UTC m=+0.461834769,LastTimestamp:2026-06-02 15:46:54.777286843 +0000 UTC m=+0.461834769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-190.ec2.internal,}"
Jun 02 15:46:54.798404 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.798387    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qt6cw"
Jun 02 15:46:54.801223 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.801136    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-190.ec2.internal.18b54e346d80885e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-190.ec2.internal,UID:ip-10-0-135-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-190.ec2.internal,},FirstTimestamp:2026-06-02 15:46:54.777305182 +0000 UTC m=+0.461853108,LastTimestamp:2026-06-02 15:46:54.777305182 +0000 UTC m=+0.461853108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-190.ec2.internal,}"
Jun 02 15:46:54.810651 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.810638    2568 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.810664    2568 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.810676    2568 server.go:85] "Starting device plugin registration server"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.810909    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.810921    2568 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.811002    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.811176    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.811187    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.811582    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:46:54.825537 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.811612    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:54.886989 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.886959    2568 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:46:54.888060 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.888033    2568 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:46:54.888118 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.888063    2568 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:46:54.888118 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.888079    2568 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:46:54.888118 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.888086    2568 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:46:54.888239 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.888118    2568 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:46:54.892157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.892114    2568 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:54.911584 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.911571    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.912346 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.912329    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.912434 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.912365    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.912434 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.912381    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:54.912434 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.912409    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-190.ec2.internal"
Jun 02 15:46:54.926078 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.926062    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-190.ec2.internal"
Jun 02 15:46:54.926144 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.926081    2568 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-190.ec2.internal\": node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:54.956859 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:54.956842    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:54.988617 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.988593    2568 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"]
Jun 02 15:46:54.988687 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.988660    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.989449 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.989428    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.989494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.989464    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.989494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.989474    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:54.990747 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.990736    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.990899 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.990886    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:54.990936 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.990916    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.991422 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991397    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.991422 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991410    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.991422 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991424    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.991610 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991434    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.991610 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991446    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:54.991610 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.991436    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:54.992620 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.992603    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:54.992697 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.992627    2568 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:54.993214 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.993197    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:54.993276 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.993229    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:54.993276 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:54.993244    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:55.014581 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.014560    2568 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-190.ec2.internal\" not found" node="ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.018657 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.018639    2568 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-190.ec2.internal\" not found" node="ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.057415 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.057398    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.059361 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.059345    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.059416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.059383    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.059416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.059406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0408568ef573e264a6bee7a242559ac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-190.ec2.internal\" (UID: \"c0408568ef573e264a6bee7a242559ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.157798 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.157734    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.159952 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.159935    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.159998 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.159964    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.159998 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.159981    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0408568ef573e264a6bee7a242559ac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-190.ec2.internal\" (UID: \"c0408568ef573e264a6bee7a242559ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.160114 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.160007    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c0408568ef573e264a6bee7a242559ac-config\") pod \"kube-apiserver-proxy-ip-10-0-135-190.ec2.internal\" (UID: \"c0408568ef573e264a6bee7a242559ac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.160114 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.160019    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.160114 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.160045    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e44cdbfa036f3016ba83b7c2b35fbfd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal\" (UID: \"e44cdbfa036f3016ba83b7c2b35fbfd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.258182 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.258141    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.316506 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.316484    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.322413 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.322398    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.358233 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.358206    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.458627 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.458557    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.559060 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.559021    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.659453 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:55.659421    2568 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-190.ec2.internal\" not found"
Jun 02 15:46:55.661578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.661561    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:46:55.661713 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.661698    2568 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:55.741145 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.741116    2568 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:55.757541 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.757510    2568 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.757541 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.757528    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:55.772103 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.772083    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:55.774011 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.773849    2568 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal"
Jun 02 15:46:55.777797 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.777777    2568 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:55.784816 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.784798    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:55.799894 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.799867    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:41:54 +0000 UTC" deadline="2028-02-27 12:04:45.852746412 +0000 UTC"
Jun 02 15:46:55.799894 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.799890    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15236h17m50.052858524s"
Jun 02 15:46:55.807476 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.807459    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tpdr7"
Jun 02 15:46:55.814980 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.814960    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tpdr7"
Jun 02 15:46:55.830600 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:55.830568    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44cdbfa036f3016ba83b7c2b35fbfd2.slice/crio-a1aed07dbd3908d14d5d1dc44c04dcd99d6a3bf312db263a4a6a0fb37a53df80 WatchSource:0}: Error finding container a1aed07dbd3908d14d5d1dc44c04dcd99d6a3bf312db263a4a6a0fb37a53df80: Status 404 returned error can't find the container with id a1aed07dbd3908d14d5d1dc44c04dcd99d6a3bf312db263a4a6a0fb37a53df80
Jun 02 15:46:55.830809 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:55.830790    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0408568ef573e264a6bee7a242559ac.slice/crio-27e71f21100036c8ceeb5f873a73d6e84d991bb157068da764ba042a4cb5471a WatchSource:0}: Error finding container 27e71f21100036c8ceeb5f873a73d6e84d991bb157068da764ba042a4cb5471a: Status 404 returned error can't find the container with id 27e71f21100036c8ceeb5f873a73d6e84d991bb157068da764ba042a4cb5471a
Jun 02 15:46:55.834148 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.834133    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:46:55.891020 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.890973    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal" event={"ID":"e44cdbfa036f3016ba83b7c2b35fbfd2","Type":"ContainerStarted","Data":"a1aed07dbd3908d14d5d1dc44c04dcd99d6a3bf312db263a4a6a0fb37a53df80"}
Jun 02 15:46:55.891774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.891748    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal" event={"ID":"c0408568ef573e264a6bee7a242559ac","Type":"ContainerStarted","Data":"27e71f21100036c8ceeb5f873a73d6e84d991bb157068da764ba042a4cb5471a"}
Jun 02 15:46:55.899730 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:55.899710    2568 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:56.185747 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.185717    2568 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:56.736940 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.736903    2568 apiserver.go:52] "Watching apiserver"
Jun 02 15:46:56.742984 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.742961    2568 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:46:56.743385 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.743355    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal","openshift-multus/multus-additional-cni-plugins-knr5w","openshift-multus/multus-x2vql","openshift-network-diagnostics/network-check-target-69vh8","openshift-network-operator/iptables-alerter-tlm7d","kube-system/global-pull-secret-syncer-xfr2t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg","openshift-dns/node-resolver-gltw9","openshift-image-registry/node-ca-tqbr4","openshift-multus/network-metrics-daemon-xtvfm","openshift-ovn-kubernetes/ovnkube-node-n79m7","kube-system/konnectivity-agent-pwm24","kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5ltb8"]
Jun 02 15:46:56.744808 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.744792    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.745886 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.745867    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.746913 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.746895    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.747005 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.746895    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.747327 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.747309    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.747416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.747337    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn2ph\""
Jun 02 15:46:56.748016 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748002    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:46:56.748457 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748296    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.748457 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748313    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.748874 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748728    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:46:56.748874 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748858    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:46:56.749011 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748943    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.749011 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.748957    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:46:56.749129 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.749088    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s5vwk\""
Jun 02 15:46:56.749782 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.749762    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p77xz\""
Jun 02 15:46:56.749882 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.749796    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.749882 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.749866    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:46:56.750020 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.750005    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.750245 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.750227    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.750467 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.750449    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vsw8b\""
Jun 02 15:46:56.751086 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.751070    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.752246 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.752229    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:56.752353 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.752284    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:46:56.753365 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.753338    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.754398 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.754290    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:46:56.754398 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.754300    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.754398 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.754372    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.754606 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.754429    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lqclj\""
Jun 02 15:46:56.754920 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.754903    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.755382 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.755363    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.755382 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.755363    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gnmp6\""
Jun 02 15:46:56.755527 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.755386    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.755527 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.755380    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:46:56.756653 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.756636    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.757314 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.757295    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.757578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.757557    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.757703 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.757681    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:46:56.757826 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.757808    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xcvq4\""
Jun 02 15:46:56.758200 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.758184    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.758774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.758757    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:46:56.760549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.759661    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:46:56.760549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.759775    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:46:56.760549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.759928    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9w8fp\""
Jun 02 15:46:56.760549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.759961    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:46:56.760549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.760329    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:46:56.760816 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.760743    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.761937 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.761317    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:46:56.762236 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.762217    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:46:56.762964 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.762675    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cnjs\""
Jun 02 15:46:56.762964 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.762917    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:46:56.764145 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.764123    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:46:56.768538 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.768632 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.768632 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768566    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-multus\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.768632 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8285q\" (UniqueName: \"kubernetes.io/projected/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-kube-api-access-8285q\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.768791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768647    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-sys\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.768791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768697    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-device-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.768791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.768791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768759    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-system-cni-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.768791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768783    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-cnibin\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-host-slash\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-systemd\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768882    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-kubelet\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-ovn\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768930    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-system-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768954    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.768978    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cni-binary-copy\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769019    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-etc-kubernetes\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-tmp\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769099    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099e71f1-a837-487b-a2e1-70e9e992800a-hosts-file\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysconfig\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769255    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-log-socket\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769278    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769312    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-config\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769333 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769326    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-script-lib\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769343    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b470d3-1650-4994-9a8d-351b7ce6f3c5-agent-certs\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-hostroot\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97jq\" (UniqueName: \"kubernetes.io/projected/0e6bad57-9fda-42cd-9570-a90fec836696-kube-api-access-p97jq\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72vp\" (UniqueName: \"kubernetes.io/projected/e07b584a-2b02-442f-80f5-e5d7d89f1452-kube-api-access-d72vp\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-socket-dir-parent\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-netns\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ba1511-d074-4216-ad97-1459127915d0-host\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769543    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-run\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-lib-modules\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-netd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-os-release\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72ba1511-d074-4216-ad97-1459127915d0-serviceca\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvxq\" (UniqueName: \"kubernetes.io/projected/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-kube-api-access-hwvxq\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769684    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprhv\" (UniqueName: \"kubernetes.io/projected/ecebf192-d529-4589-abcb-89f1f8a13ea0-kube-api-access-jprhv\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.769735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-tuned\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769759    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx9j\" (UniqueName: \"kubernetes.io/projected/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-kube-api-access-2kx9j\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769794    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769850    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769880    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-conf\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769913    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-systemd-units\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769932    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-netns\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769948    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-etc-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769966    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b470d3-1650-4994-9a8d-351b7ce6f3c5-konnectivity-ca\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.769991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770012    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-bin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnts6\" (UniqueName: \"kubernetes.io/projected/72ba1511-d074-4216-ad97-1459127915d0-kube-api-access-rnts6\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-host\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6bad57-9fda-42cd-9570-a90fec836696-ovn-node-metrics-cert\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-binary-copy\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.770376 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770176    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099e71f1-a837-487b-a2e1-70e9e992800a-tmp-dir\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-var-lib-kubelet\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-os-release\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770256    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-k8s-cni-cncf-io\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770271    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-multus-certs\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770285    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-kubernetes\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770304    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-bin\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cnibin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhzd\" (UniqueName: \"kubernetes.io/projected/c0186ca7-bafe-4ba4-82bf-0257fbff4133-kube-api-access-jmhzd\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770390    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-iptables-alerter-script\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-systemd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770432    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-slash\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770448    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770469    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-conf-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770504    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-var-lib-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770535    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-node-log\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771074 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-env-overrides\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.771715 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770606    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-daemon-config\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.771715 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.771715 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2qd\" (UniqueName: \"kubernetes.io/projected/099e71f1-a837-487b-a2e1-70e9e992800a-kube-api-access-nh2qd\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.771715 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770688    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-modprobe-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.771715 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.770711    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-kubelet\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.815964 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.815928    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:55 +0000 UTC" deadline="2028-01-02 13:58:52.124642013 +0000 UTC"
Jun 02 15:46:56.815964 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.815962    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13894h11m55.308683909s"
Jun 02 15:46:56.858984 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.858950    2568 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:46:56.870990 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.870957    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-binary-copy\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.871172 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871000    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099e71f1-a837-487b-a2e1-70e9e992800a-tmp-dir\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.871172 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871031    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-var-lib-kubelet\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.871172 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871074    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-os-release\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.871172 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871116    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-k8s-cni-cncf-io\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871172 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871144    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-multus-certs\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871179    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-kubelet-config\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871186    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-var-lib-kubelet\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871214    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-kubernetes\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871240    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-bin\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871268    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-k8s-cni-cncf-io\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871275    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cnibin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871307    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhzd\" (UniqueName: \"kubernetes.io/projected/c0186ca7-bafe-4ba4-82bf-0257fbff4133-kube-api-access-jmhzd\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.871425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871380    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-bin\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871452    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-os-release\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871498    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-kubernetes\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871533    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-multus-certs\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871570    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-iptables-alerter-script\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871609    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-binary-copy\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871621    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-systemd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871647    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-slash\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871677    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cnibin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871689    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871719    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-systemd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871722    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-conf-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.871791 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871753    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-slash\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871909    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-var-lib-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.871929    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-conf-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872000    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-node-log\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872032    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-var-lib-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872082    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/099e71f1-a837-487b-a2e1-70e9e992800a-tmp-dir\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872093    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-env-overrides\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872097    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-node-log\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872148    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-daemon-config\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872194    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.872412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872349    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-iptables-alerter-script\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872558    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-env-overrides\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872695    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872727    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2qd\" (UniqueName: \"kubernetes.io/projected/099e71f1-a837-487b-a2e1-70e9e992800a-kube-api-access-nh2qd\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872750    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-daemon-config\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872756    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-modprobe-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872779    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-kubelet\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.872804 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872800    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-dbus\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872815    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872832    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872856    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-multus\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872898    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-modprobe-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872906    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872901    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-multus\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.872922    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872928    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.872983    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8285q\" (UniqueName: \"kubernetes.io/projected/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-kube-api-access-8285q\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873039    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-kubelet\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.873094 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.873081    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.373040144 +0000 UTC m=+3.057588076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873102    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-sys\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873127    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-device-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873151    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873176    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-system-cni-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873187    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-sys\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873199    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-cnibin\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873223    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-host-slash\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873220    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-device-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873248    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-systemd\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873265    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-system-cni-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873303    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-host-slash\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873307    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873312    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-kubelet\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873305    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-systemd\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873330    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-cnibin\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873341    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-ovn\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.873570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873362    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-run-ovn\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873361    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-system-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873389    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-kubelet\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873417    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-system-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873406    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873463    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-cni-dir\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873460    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cni-binary-copy\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873501    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-etc-kubernetes\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873528    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-tmp\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873555    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873581    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873605    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873662    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-etc-kubernetes\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873670    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099e71f1-a837-487b-a2e1-70e9e992800a-hosts-file\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873699    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysconfig\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873712    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873725    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873723    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-log-socket\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.874349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873757    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/099e71f1-a837-487b-a2e1-70e9e992800a-hosts-file\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873765    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873786    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysconfig\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873793    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-log-socket\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873792    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-config\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873830    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873841    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-script-lib\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873860    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873886    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b470d3-1650-4994-9a8d-351b7ce6f3c5-agent-certs\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873911    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-cni-binary-copy\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873938    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-hostroot\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.873972    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p97jq\" (UniqueName: \"kubernetes.io/projected/0e6bad57-9fda-42cd-9570-a90fec836696-kube-api-access-p97jq\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874004    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d72vp\" (UniqueName: \"kubernetes.io/projected/e07b584a-2b02-442f-80f5-e5d7d89f1452-kube-api-access-d72vp\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874030    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-socket-dir-parent\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874070    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-netns\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874095    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ba1511-d074-4216-ad97-1459127915d0-host\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874121    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-run\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874146    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-lib-modules\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874172    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-netd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874198    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874223    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-os-release\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874247    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72ba1511-d074-4216-ad97-1459127915d0-serviceca\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874275    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-run-netns\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874272    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvxq\" (UniqueName: \"kubernetes.io/projected/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-kube-api-access-hwvxq\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874313    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-config\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874324    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jprhv\" (UniqueName: \"kubernetes.io/projected/ecebf192-d529-4589-abcb-89f1f8a13ea0-kube-api-access-jprhv\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874369    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-cni-netd\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874398    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-tuned\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874423    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx9j\" (UniqueName: \"kubernetes.io/projected/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-kube-api-access-2kx9j\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874448    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874457    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0e6bad57-9fda-42cd-9570-a90fec836696-ovnkube-script-lib\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874471    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874515    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-run\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874550    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-conf\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.875992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874549    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e07b584a-2b02-442f-80f5-e5d7d89f1452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874598    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ba1511-d074-4216-ad97-1459127915d0-host\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874620    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-multus-socket-dir-parent\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874625    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-systemd-units\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874656    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-netns\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874658    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-systemd-units\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874663    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874682    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-etc-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874703    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ecebf192-d529-4589-abcb-89f1f8a13ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874708    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b470d3-1650-4994-9a8d-351b7ce6f3c5-konnectivity-ca\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874712    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-hostroot\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874728    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-host-run-netns\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874735    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874759    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-bin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874769    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-os-release\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874780    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rnts6\" (UniqueName: \"kubernetes.io/projected/72ba1511-d074-4216-ad97-1459127915d0-kube-api-access-rnts6\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874802    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874811    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0e6bad57-9fda-42cd-9570-a90fec836696-etc-openvswitch\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.876853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874815    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-conf\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874825    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-host\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874856    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-host\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874870    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6bad57-9fda-42cd-9570-a90fec836696-ovn-node-metrics-cert\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874905    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-lib-modules\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.874967    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-host-var-lib-cni-bin\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.875197    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72ba1511-d074-4216-ad97-1459127915d0-serviceca\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.875362    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19b470d3-1650-4994-9a8d-351b7ce6f3c5-konnectivity-ca\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.875375    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-sysctl-d\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.877639 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.875390    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e07b584a-2b02-442f-80f5-e5d7d89f1452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.880668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.878287    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e6bad57-9fda-42cd-9570-a90fec836696-ovn-node-metrics-cert\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.880668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.878322    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-tmp\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.880668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.878393    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19b470d3-1650-4994-9a8d-351b7ce6f3c5-agent-certs\") pod \"konnectivity-agent-pwm24\" (UID: \"19b470d3-1650-4994-9a8d-351b7ce6f3c5\") " pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:56.880668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.878423    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-etc-tuned\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.882963 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.882942    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhzd\" (UniqueName: \"kubernetes.io/projected/c0186ca7-bafe-4ba4-82bf-0257fbff4133-kube-api-access-jmhzd\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:56.885717 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.885694    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2qd\" (UniqueName: \"kubernetes.io/projected/099e71f1-a837-487b-a2e1-70e9e992800a-kube-api-access-nh2qd\") pod \"node-resolver-gltw9\" (UID: \"099e71f1-a837-487b-a2e1-70e9e992800a\") " pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:56.889582 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.889552    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:56.889582 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.889584    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:56.889753 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.889598    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:56.889753 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.889672    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.389652819 +0000 UTC m=+3.074200759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:56.891519 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.891492    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvxq\" (UniqueName: \"kubernetes.io/projected/72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1-kube-api-access-hwvxq\") pod \"iptables-alerter-tlm7d\" (UID: \"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1\") " pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:56.893473 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.893413    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprhv\" (UniqueName: \"kubernetes.io/projected/ecebf192-d529-4589-abcb-89f1f8a13ea0-kube-api-access-jprhv\") pod \"aws-ebs-csi-driver-node-8rmrg\" (UID: \"ecebf192-d529-4589-abcb-89f1f8a13ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:56.894359 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.894063    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97jq\" (UniqueName: \"kubernetes.io/projected/0e6bad57-9fda-42cd-9570-a90fec836696-kube-api-access-p97jq\") pod \"ovnkube-node-n79m7\" (UID: \"0e6bad57-9fda-42cd-9570-a90fec836696\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:56.894845 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.894820    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8285q\" (UniqueName: \"kubernetes.io/projected/10fc7050-27b5-48fa-9b5b-5260a5bedf0a-kube-api-access-8285q\") pod \"multus-x2vql\" (UID: \"10fc7050-27b5-48fa-9b5b-5260a5bedf0a\") " pod="openshift-multus/multus-x2vql"
Jun 02 15:46:56.895169 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.895147    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx9j\" (UniqueName: \"kubernetes.io/projected/d3c0a55d-d692-4cc6-8551-d043fd89a9c5-kube-api-access-2kx9j\") pod \"tuned-5ltb8\" (UID: \"d3c0a55d-d692-4cc6-8551-d043fd89a9c5\") " pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:56.896475 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.896458    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnts6\" (UniqueName: \"kubernetes.io/projected/72ba1511-d074-4216-ad97-1459127915d0-kube-api-access-rnts6\") pod \"node-ca-tqbr4\" (UID: \"72ba1511-d074-4216-ad97-1459127915d0\") " pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:56.896475 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.896466    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72vp\" (UniqueName: \"kubernetes.io/projected/e07b584a-2b02-442f-80f5-e5d7d89f1452-kube-api-access-d72vp\") pod \"multus-additional-cni-plugins-knr5w\" (UID: \"e07b584a-2b02-442f-80f5-e5d7d89f1452\") " pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:56.975561 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.975523    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-dbus\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.975731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.975605    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.975731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.975642    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-kubelet-config\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.975731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.975718    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-kubelet-config\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.975867 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:56.975729    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-dbus\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:56.975867 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.975762    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:56.975867 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:56.975838    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.475816583 +0000 UTC m=+3.160364496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:57.057354 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.057271    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gltw9"
Jun 02 15:46:57.066593 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.066571    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knr5w"
Jun 02 15:46:57.075286 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.075268    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:46:57.080949 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.080930    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2vql"
Jun 02 15:46:57.088447 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.088424    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5ltb8"
Jun 02 15:46:57.094958 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.094937    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tqbr4"
Jun 02 15:46:57.101525 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.101507    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg"
Jun 02 15:46:57.109063 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.109030    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlm7d"
Jun 02 15:46:57.113660 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.113642    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:46:57.218204 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.218172    2568 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:57.333958 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.333931    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b470d3_1650_4994_9a8d_351b7ce6f3c5.slice/crio-7889c6902794cf05890fd46fcfc449deb532c28a817b53414b8c8e5698ce75a2 WatchSource:0}: Error finding container 7889c6902794cf05890fd46fcfc449deb532c28a817b53414b8c8e5698ce75a2: Status 404 returned error can't find the container with id 7889c6902794cf05890fd46fcfc449deb532c28a817b53414b8c8e5698ce75a2
Jun 02 15:46:57.335532 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.335504    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c0a55d_d692_4cc6_8551_d043fd89a9c5.slice/crio-295b7e2f427c8f8c6cac4fa987ee91c5604b3497312b442268ccc62b19915a98 WatchSource:0}: Error finding container 295b7e2f427c8f8c6cac4fa987ee91c5604b3497312b442268ccc62b19915a98: Status 404 returned error can't find the container with id 295b7e2f427c8f8c6cac4fa987ee91c5604b3497312b442268ccc62b19915a98
Jun 02 15:46:57.336557 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.336527    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecebf192_d529_4589_abcb_89f1f8a13ea0.slice/crio-c8dbff4a24308f787a26ba4bb33b72f48b18435d17550ffb48f2e582cf61572e WatchSource:0}: Error finding container c8dbff4a24308f787a26ba4bb33b72f48b18435d17550ffb48f2e582cf61572e: Status 404 returned error can't find the container with id c8dbff4a24308f787a26ba4bb33b72f48b18435d17550ffb48f2e582cf61572e
Jun 02 15:46:57.341478 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.341455    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6bad57_9fda_42cd_9570_a90fec836696.slice/crio-9903d0cf87721fca29e77291ec298a6448bca4a6a40238224198097913ab8f2d WatchSource:0}: Error finding container 9903d0cf87721fca29e77291ec298a6448bca4a6a40238224198097913ab8f2d: Status 404 returned error can't find the container with id 9903d0cf87721fca29e77291ec298a6448bca4a6a40238224198097913ab8f2d
Jun 02 15:46:57.342529 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.342494    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fc7050_27b5_48fa_9b5b_5260a5bedf0a.slice/crio-416bd769d098cffcc21c5cf27371b38ab7e8fd2c57806bc21e7b28f13ceeec1d WatchSource:0}: Error finding container 416bd769d098cffcc21c5cf27371b38ab7e8fd2c57806bc21e7b28f13ceeec1d: Status 404 returned error can't find the container with id 416bd769d098cffcc21c5cf27371b38ab7e8fd2c57806bc21e7b28f13ceeec1d
Jun 02 15:46:57.343374 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.343355    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cf9dd7_8c06_446d_bf22_51ddb1b7d3c1.slice/crio-d21ca6785bd60d2a118e05e0afce32af4d72172918217b7188d3085d0d4ef5d6 WatchSource:0}: Error finding container d21ca6785bd60d2a118e05e0afce32af4d72172918217b7188d3085d0d4ef5d6: Status 404 returned error can't find the container with id d21ca6785bd60d2a118e05e0afce32af4d72172918217b7188d3085d0d4ef5d6
Jun 02 15:46:57.344468 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.344435    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ba1511_d074_4216_ad97_1459127915d0.slice/crio-8b1f52c2c0c9ca589e9305ddd71ab43ca16d9d54dab978ea36134c7943a10616 WatchSource:0}: Error finding container 8b1f52c2c0c9ca589e9305ddd71ab43ca16d9d54dab978ea36134c7943a10616: Status 404 returned error can't find the container with id 8b1f52c2c0c9ca589e9305ddd71ab43ca16d9d54dab978ea36134c7943a10616
Jun 02 15:46:57.345095 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.345074    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099e71f1_a837_487b_a2e1_70e9e992800a.slice/crio-fa0c7889298a8e6362c0da060489220a4dcf9a4b2e193a79a047e52b38a90f29 WatchSource:0}: Error finding container fa0c7889298a8e6362c0da060489220a4dcf9a4b2e193a79a047e52b38a90f29: Status 404 returned error can't find the container with id fa0c7889298a8e6362c0da060489220a4dcf9a4b2e193a79a047e52b38a90f29
Jun 02 15:46:57.346495 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:46:57.346471    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07b584a_2b02_442f_80f5_e5d7d89f1452.slice/crio-bfce6bca9d01e93bd88abe6216f1dc3ce007752d699b4ae1a795eea7a3761699 WatchSource:0}: Error finding container bfce6bca9d01e93bd88abe6216f1dc3ce007752d699b4ae1a795eea7a3761699: Status 404 returned error can't find the container with id bfce6bca9d01e93bd88abe6216f1dc3ce007752d699b4ae1a795eea7a3761699
Jun 02 15:46:57.378366 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.378344    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:57.378455 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.378436    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:57.378500 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.378479    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:58.378467542 +0000 UTC m=+4.063015455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:57.478878 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.478839    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.478895    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.478964    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.478973    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.478983    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.478994    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:57.479023 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.479027    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:46:58.479013404 +0000 UTC m=+4.163561316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:57.479309 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:57.479041    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:46:58.479035049 +0000 UTC m=+4.163582962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:57.816570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.816520    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:55 +0000 UTC" deadline="2028-02-23 21:55:42.316550198 +0000 UTC"
Jun 02 15:46:57.816570 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.816558    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15150h8m44.499996594s"
Jun 02 15:46:57.900389 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.900319    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerStarted","Data":"bfce6bca9d01e93bd88abe6216f1dc3ce007752d699b4ae1a795eea7a3761699"}
Jun 02 15:46:57.905724 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.905681    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tqbr4" event={"ID":"72ba1511-d074-4216-ad97-1459127915d0","Type":"ContainerStarted","Data":"8b1f52c2c0c9ca589e9305ddd71ab43ca16d9d54dab978ea36134c7943a10616"}
Jun 02 15:46:57.909456 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.909420    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2vql" event={"ID":"10fc7050-27b5-48fa-9b5b-5260a5bedf0a","Type":"ContainerStarted","Data":"416bd769d098cffcc21c5cf27371b38ab7e8fd2c57806bc21e7b28f13ceeec1d"}
Jun 02 15:46:57.919662 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.919623    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"9903d0cf87721fca29e77291ec298a6448bca4a6a40238224198097913ab8f2d"}
Jun 02 15:46:57.923550 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.923523    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5ltb8" event={"ID":"d3c0a55d-d692-4cc6-8551-d043fd89a9c5","Type":"ContainerStarted","Data":"295b7e2f427c8f8c6cac4fa987ee91c5604b3497312b442268ccc62b19915a98"}
Jun 02 15:46:57.931673 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.931643    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gltw9" event={"ID":"099e71f1-a837-487b-a2e1-70e9e992800a","Type":"ContainerStarted","Data":"fa0c7889298a8e6362c0da060489220a4dcf9a4b2e193a79a047e52b38a90f29"}
Jun 02 15:46:57.936625 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.936597    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlm7d" event={"ID":"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1","Type":"ContainerStarted","Data":"d21ca6785bd60d2a118e05e0afce32af4d72172918217b7188d3085d0d4ef5d6"}
Jun 02 15:46:57.941866 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.941841    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg" event={"ID":"ecebf192-d529-4589-abcb-89f1f8a13ea0","Type":"ContainerStarted","Data":"c8dbff4a24308f787a26ba4bb33b72f48b18435d17550ffb48f2e582cf61572e"}
Jun 02 15:46:57.944689 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.944660    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwm24" event={"ID":"19b470d3-1650-4994-9a8d-351b7ce6f3c5","Type":"ContainerStarted","Data":"7889c6902794cf05890fd46fcfc449deb532c28a817b53414b8c8e5698ce75a2"}
Jun 02 15:46:57.952104 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.952077    2568 generic.go:358] "Generic (PLEG): container finished" podID="e44cdbfa036f3016ba83b7c2b35fbfd2" containerID="da0a928b6ade807414702ece32563573624addb3d3725c1db6ceb583f941e695" exitCode=0
Jun 02 15:46:57.952196 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.952151    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal" event={"ID":"e44cdbfa036f3016ba83b7c2b35fbfd2","Type":"ContainerDied","Data":"da0a928b6ade807414702ece32563573624addb3d3725c1db6ceb583f941e695"}
Jun 02 15:46:57.967174 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.967149    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal" event={"ID":"c0408568ef573e264a6bee7a242559ac","Type":"ContainerStarted","Data":"68d28a4bf918ada7fd7f53a1a0b0e31dea65d59045f10c0ed34d1674e0233b25"}
Jun 02 15:46:57.983585 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:57.983536    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-190.ec2.internal" podStartSLOduration=2.983519624 podStartE2EDuration="2.983519624s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:57.9833339 +0000 UTC m=+3.667881838" watchObservedRunningTime="2026-06-02 15:46:57.983519624 +0000 UTC m=+3.668067592"
Jun 02 15:46:58.387503 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.387467    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:58.387667 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.387627    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:58.387738 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.387690    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:00.387671389 +0000 UTC m=+6.072219306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:58.488283 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.488201    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:58.488283 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.488258    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:58.488503 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488411    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:58.488503 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488476    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:47:00.488457553 +0000 UTC m=+6.173005485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:58.488942 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488922    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:58.488942 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488945    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:58.489114 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488958    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:58.489114 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.488999    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:00.4889846 +0000 UTC m=+6.173532527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:58.889245 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.889219    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:46:58.889682 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.889341    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:46:58.889900 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.889879    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:46:58.890014 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.889984    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:46:58.892418 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:58.892398    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:46:58.892517 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:46:58.892494    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:46:59.010731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:59.010697    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal" event={"ID":"e44cdbfa036f3016ba83b7c2b35fbfd2","Type":"ContainerStarted","Data":"10b971f3bef8b5d327cc66334ef5c7b7acc2b0026d53b6e84fc5c25e0305530e"}
Jun 02 15:46:59.024786 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:46:59.024733    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-190.ec2.internal" podStartSLOduration=4.02471347 podStartE2EDuration="4.02471347s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:59.023431974 +0000 UTC m=+4.707979909" watchObservedRunningTime="2026-06-02 15:46:59.02471347 +0000 UTC m=+4.709261406"
Jun 02 15:47:00.405989 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.405839    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:00.405989 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.405985    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:00.406490 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.406068    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:04.406031063 +0000 UTC m=+10.090578989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:00.506392 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.506349    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:00.506569 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.506423    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:00.506632 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506582    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:00.506691 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506672    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:00.506743 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506698    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:00.506743 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506711    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:00.506831 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506681    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:47:04.506661617 +0000 UTC m=+10.191209535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:00.506831 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.506775    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:04.506759278 +0000 UTC m=+10.191307206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.888998    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.888998    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:00.888998    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.889162    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.889601    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:00.889714 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:00.889681    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:02.889036 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:02.888981    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:02.889036 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:02.889028    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:02.889567 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:02.889130    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:02.889567 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:02.889206    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:02.889567 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:02.889265    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:02.889567 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:02.889341    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:04.439577 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.439539    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:04.440067 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.439676    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:04.440067 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.439747    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:12.439727161 +0000 UTC m=+18.124275079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:04.539955 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.539914    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:04.540159 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.540003    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:04.540159 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540154    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:04.540268 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540185    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:04.540268 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540185    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:04.540268 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540199    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:04.540268 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540262    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:47:12.540241383 +0000 UTC m=+18.224789310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:04.540462 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.540281    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:12.540271818 +0000 UTC m=+18.224819739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:04.889400 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.889359    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:04.889400 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.889359    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:04.890488 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:04.890463    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:04.890629 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.890590    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:04.891109 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.891000    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:04.891203 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:04.891104    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:06.889035 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:06.889000    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:06.889035 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:06.889023    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:06.889035 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:06.889000    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:06.889610 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:06.889153    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:06.889610 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:06.889231    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:06.889610 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:06.889280    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:08.888722 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:08.888690    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:08.889220 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:08.888800    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:08.889220 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:08.888808    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:08.889220 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:08.888945    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:08.889220 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:08.888975    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:08.889220 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:08.889101    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:10.889284 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:10.889253    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:10.889284 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:10.889277    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:10.889801 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:10.889373    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:10.889801 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:10.889400    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:10.889801 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:10.889505    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:10.889801 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:10.889604    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:12.500895 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.500855    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:12.501333 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.500991    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:12.501333 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.501067    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.501038001 +0000 UTC m=+34.185585914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:12.601716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.601677    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:12.601882 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.601744    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:12.601882 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.601841    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:12.601976 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.601906    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.601892124 +0000 UTC m=+34.286440037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:12.601976 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.601842    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:12.602121 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.601979    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:12.602121 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.601998    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:12.602121 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.602081    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.602064636 +0000 UTC m=+34.286612563 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:12.888349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.888299    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:12.888349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.888325    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:12.888581 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.888435    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:12.888581 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.888506    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:12.888581 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:12.888542    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:12.888720 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:12.888647    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:14.889679 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:14.889648    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:14.890248 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:14.889744    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:14.890248 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:14.889829    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:14.890248 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:14.889939    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:14.890248 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:14.889988    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:14.890248 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:14.890071    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:16.039817 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.039519    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwm24" event={"ID":"19b470d3-1650-4994-9a8d-351b7ce6f3c5","Type":"ContainerStarted","Data":"7dd19cd5d535312123d1d673b1fe0bdff37b26e7df83bf7ba361ed1ac6b151af"}
Jun 02 15:47:16.041148 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.041112    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="c1eb8bbaa027e585d89fe3f91e5057a97a4ee0f36dc929287130aecbe567a708" exitCode=0
Jun 02 15:47:16.041233 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.041198    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"c1eb8bbaa027e585d89fe3f91e5057a97a4ee0f36dc929287130aecbe567a708"}
Jun 02 15:47:16.043188 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.043144    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tqbr4" event={"ID":"72ba1511-d074-4216-ad97-1459127915d0","Type":"ContainerStarted","Data":"4221fa75f5f7c60518bea5848a349be53469c3a1ce99cad8ca4e7a6659bc7445"}
Jun 02 15:47:16.044761 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.044730    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2vql" event={"ID":"10fc7050-27b5-48fa-9b5b-5260a5bedf0a","Type":"ContainerStarted","Data":"ae11ce3c4625b85b939e0b83e13850ea95df1a5179903c734e707ac74f0e5fb4"}
Jun 02 15:47:16.047302 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047286    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:47:16.047562 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047545    2568 generic.go:358] "Generic (PLEG): container finished" podID="0e6bad57-9fda-42cd-9570-a90fec836696" containerID="e525660ab21541c8c7879c0121b3d7b71961354d7caf32f81928c5072ab9f6ae" exitCode=1
Jun 02 15:47:16.047638 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047599    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"a41e0f88645b599de899e9de539d27dd85d4e3ca8e854a3cd674c0de3cb630b0"}
Jun 02 15:47:16.047638 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047619    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"bbfc772eebae814ffd424009c36192e330da67c83e1c2348dd0f9e5a5977fd5a"}
Jun 02 15:47:16.047784 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047631    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"4d0a1f4f161e200f8394f8b06c77c264a6eb49532e466f41cbc657518dc07f81"}
Jun 02 15:47:16.047784 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047654    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerDied","Data":"e525660ab21541c8c7879c0121b3d7b71961354d7caf32f81928c5072ab9f6ae"}
Jun 02 15:47:16.047784 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.047667    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"46c1365ffadeef4bcbfc3bd2174ade896cff2e14ead74b1646c297503019e260"}
Jun 02 15:47:16.048746 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.048705    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5ltb8" event={"ID":"d3c0a55d-d692-4cc6-8551-d043fd89a9c5","Type":"ContainerStarted","Data":"1c8e508284bbdd76f713d142b0aee9c55b9090632653f74c4ef632225c7a6bff"}
Jun 02 15:47:16.050005 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.049979    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gltw9" event={"ID":"099e71f1-a837-487b-a2e1-70e9e992800a","Type":"ContainerStarted","Data":"255db6293181fc89e2ee1010284f281cd894e542e82f4aa57a4d444f39a5bcb9"}
Jun 02 15:47:16.051929 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.051908    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg" event={"ID":"ecebf192-d529-4589-abcb-89f1f8a13ea0","Type":"ContainerStarted","Data":"b6730030bea380ffa5076ede0cce8a28e47bd7ca392aa37e6c6b86befa023275"}
Jun 02 15:47:16.082192 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.082139    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pwm24" podStartSLOduration=11.172813446 podStartE2EDuration="21.082124178s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.33563826 +0000 UTC m=+3.020186399" lastFinishedPulling="2026-06-02 15:47:07.244949204 +0000 UTC m=+12.929497131" observedRunningTime="2026-06-02 15:47:16.05780832 +0000 UTC m=+21.742356258" watchObservedRunningTime="2026-06-02 15:47:16.082124178 +0000 UTC m=+21.766672113"
Jun 02 15:47:16.105675 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.105635    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-gltw9" podStartSLOduration=4.259920415 podStartE2EDuration="22.105622193s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.347318921 +0000 UTC m=+3.031866848" lastFinishedPulling="2026-06-02 15:47:15.193020713 +0000 UTC m=+20.877568626" observedRunningTime="2026-06-02 15:47:16.082734832 +0000 UTC m=+21.767282766" watchObservedRunningTime="2026-06-02 15:47:16.105622193 +0000 UTC m=+21.790170127"
Jun 02 15:47:16.136812 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.136771    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5ltb8" podStartSLOduration=4.244036721 podStartE2EDuration="22.136756847s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.338510391 +0000 UTC m=+3.023058308" lastFinishedPulling="2026-06-02 15:47:15.231230508 +0000 UTC m=+20.915778434" observedRunningTime="2026-06-02 15:47:16.136637487 +0000 UTC m=+21.821185422" watchObservedRunningTime="2026-06-02 15:47:16.136756847 +0000 UTC m=+21.821304778"
Jun 02 15:47:16.136992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.136970    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-x2vql" podStartSLOduration=4.229253097 podStartE2EDuration="22.136965198s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.344558847 +0000 UTC m=+3.029106759" lastFinishedPulling="2026-06-02 15:47:15.252270946 +0000 UTC m=+20.936818860" observedRunningTime="2026-06-02 15:47:16.106673 +0000 UTC m=+21.791220935" watchObservedRunningTime="2026-06-02 15:47:16.136965198 +0000 UTC m=+21.821513133"
Jun 02 15:47:16.166877 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.166826    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tqbr4" podStartSLOduration=4.283571381 podStartE2EDuration="22.166809457s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.346322264 +0000 UTC m=+3.030870178" lastFinishedPulling="2026-06-02 15:47:15.229560323 +0000 UTC m=+20.914108254" observedRunningTime="2026-06-02 15:47:16.162854052 +0000 UTC m=+21.847401987" watchObservedRunningTime="2026-06-02 15:47:16.166809457 +0000 UTC m=+21.851357405"
Jun 02 15:47:16.605449 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.605376    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:47:16.821240 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.821117    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:47:16.605397051Z","UUID":"94b61ab9-dbd1-4039-8bc6-0774de820b5e","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:47:16.823060 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.823032    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:47:16.823190 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.823077    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:47:16.889328 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.889301    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:16.889328 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.889326    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:16.889579 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:16.889301    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:16.889579 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:16.889411    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:16.889579 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:16.889494    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:16.889579 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:16.889567    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:17.057673 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:17.057643    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:47:17.058251 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:17.058087    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"4984e08e80247ce26f263d7ade82912545340fade83fa907d6cccac248d4ee22"}
Jun 02 15:47:17.059922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:17.059895    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlm7d" event={"ID":"72cf9dd7-8c06-446d-bf22-51ddb1b7d3c1","Type":"ContainerStarted","Data":"a9b86c02ac04b81845708d7184bf90583111525bf14a744bd33ac50d1b3eadc8"}
Jun 02 15:47:17.062250 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:17.062197    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg" event={"ID":"ecebf192-d529-4589-abcb-89f1f8a13ea0","Type":"ContainerStarted","Data":"240c1d2861b566a6f46b84f69bf811f1da12648cd0a67c76a17cfed61d2fea4f"}
Jun 02 15:47:17.081934 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:17.081761    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tlm7d" podStartSLOduration=4.234004765 podStartE2EDuration="22.081746408s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.345277182 +0000 UTC m=+3.029825102" lastFinishedPulling="2026-06-02 15:47:15.193018817 +0000 UTC m=+20.877566745" observedRunningTime="2026-06-02 15:47:17.081281905 +0000 UTC m=+22.765829843" watchObservedRunningTime="2026-06-02 15:47:17.081746408 +0000 UTC m=+22.766294344"
Jun 02 15:47:18.066406 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.066314    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg" event={"ID":"ecebf192-d529-4589-abcb-89f1f8a13ea0","Type":"ContainerStarted","Data":"5cf741ea8b26ceac65faee841e491ddb3363f0efbf22626df33f6820402ade29"}
Jun 02 15:47:18.116288 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.116226    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rmrg" podStartSLOduration=2.671410988 podStartE2EDuration="23.116207793s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.340554036 +0000 UTC m=+3.025101966" lastFinishedPulling="2026-06-02 15:47:17.785350844 +0000 UTC m=+23.469898771" observedRunningTime="2026-06-02 15:47:18.116162005 +0000 UTC m=+23.800709939" watchObservedRunningTime="2026-06-02 15:47:18.116207793 +0000 UTC m=+23.800755729"
Jun 02 15:47:18.586938 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.586727    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:47:18.587520 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.587496    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:47:18.888551 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.888513    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:18.888551 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.888544    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:18.888809 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:18.888643    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:18.888809 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:18.888674    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:18.888809 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:18.888780    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:18.888984 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:18.888865    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:19.071765 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:19.071740    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:47:19.072292 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:19.072174    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"98abab1f4f0c456009470e7cc206718a5b8acb8ba0f014456008ce37c59919c0"}
Jun 02 15:47:19.072434 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:19.072413    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:47:19.073093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:19.073072    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pwm24"
Jun 02 15:47:20.075736 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:20.075694    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerStarted","Data":"0595a5598167f9e3234531bd68b0fcea1cd0d4ff15d074ef0c23969feae72faa"}
Jun 02 15:47:20.889194 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:20.889164    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:20.889405 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:20.889164    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:20.889405 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:20.889254    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:20.889405 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:20.889175    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:20.889405 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:20.889324    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:20.889405 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:20.889384    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:21.078663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:21.078627    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="0595a5598167f9e3234531bd68b0fcea1cd0d4ff15d074ef0c23969feae72faa" exitCode=0
Jun 02 15:47:21.079305 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:21.078711    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"0595a5598167f9e3234531bd68b0fcea1cd0d4ff15d074ef0c23969feae72faa"}
Jun 02 15:47:21.081858 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:21.081835    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:47:21.082270 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:21.082248    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"4b7ad2031cf1225fdb84fc244204a09de8310b3b6c291e4d3116b5ed9366744a"}
Jun 02 15:47:21.082678 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:21.082664    2568 scope.go:117] "RemoveContainer" containerID="e525660ab21541c8c7879c0121b3d7b71961354d7caf32f81928c5072ab9f6ae"
Jun 02 15:47:22.087768 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.087708    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:47:22.088202 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.088078    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" event={"ID":"0e6bad57-9fda-42cd-9570-a90fec836696","Type":"ContainerStarted","Data":"686d3d353b34d7079e67731cc96b623c93584391374a73855d6416d2d2b3128a"}
Jun 02 15:47:22.088600 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.088436    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:47:22.088600 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.088462    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:47:22.088600 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.088484    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:47:22.090468 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.090438    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="076551bd4cb9b93e999bec6016813bfd28989815e7cff737161f76cc32840b06" exitCode=0
Jun 02 15:47:22.090553 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.090472    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"076551bd4cb9b93e999bec6016813bfd28989815e7cff737161f76cc32840b06"}
Jun 02 15:47:22.104865 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.104840    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:47:22.104991 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.104932    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:47:22.140131 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.140076    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7" podStartSLOduration=9.205625413 podStartE2EDuration="27.140042132s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.343210654 +0000 UTC m=+3.027758582" lastFinishedPulling="2026-06-02 15:47:15.277627371 +0000 UTC m=+20.962175301" observedRunningTime="2026-06-02 15:47:22.139793328 +0000 UTC m=+27.824341265" watchObservedRunningTime="2026-06-02 15:47:22.140042132 +0000 UTC m=+27.824590068"
Jun 02 15:47:22.389499 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.389468    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfr2t"]
Jun 02 15:47:22.389649 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.389599    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:22.389706 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:22.389686    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:22.395356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.395330    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69vh8"]
Jun 02 15:47:22.395458 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.395435    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:22.395524 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:22.395508    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:22.456904 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.456876    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xtvfm"]
Jun 02 15:47:22.457097 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:22.457016    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:22.457164 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:22.457147    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:23.094991 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:23.094692    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="5f6d35cf17eb46d8e5261350e9f0adcb82eda7894533c8c217d240854940a446" exitCode=0
Jun 02 15:47:23.095371 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:23.094775    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"5f6d35cf17eb46d8e5261350e9f0adcb82eda7894533c8c217d240854940a446"}
Jun 02 15:47:23.888881 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:23.888849    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:23.889036 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:23.888940    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:23.889036 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:23.889028    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:23.889179 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:23.889062    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:23.889179 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:23.889134    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:23.889295 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:23.889245    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:25.888803 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:25.888768    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:25.889416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:25.888776    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:25.889416 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:25.888884    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:25.889416 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:25.888986    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:25.889416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:25.888776    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:25.889416 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:25.889107    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:27.889301 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:27.889268    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:27.889830 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:27.889316    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:27.889830 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:27.889272    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:27.889830 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:27.889390    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xfr2t" podUID="c59b270b-1208-4a9a-bd68-2ad2507ebf2f"
Jun 02 15:47:27.889830 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:27.889494    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:47:27.889830 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:27.889576    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69vh8" podUID="2c88dfd2-651a-4626-a690-35e05cc1ba9d"
Jun 02 15:47:28.108216 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.108188    2568 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-190.ec2.internal" event="NodeReady"
Jun 02 15:47:28.108379 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.108340    2568 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:47:28.187694 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.187603    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7zqcd"]
Jun 02 15:47:28.194998 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.194967    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b98jg"]
Jun 02 15:47:28.195177 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.195154    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.197796 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.197770    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25zcr\""
Jun 02 15:47:28.197947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.197897    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:47:28.198015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.197953    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:47:28.198117 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.198098    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.200833 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.200809    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b98jg"]
Jun 02 15:47:28.203321 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.203284    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zqcd"]
Jun 02 15:47:28.205161 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.205137    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gg4df\""
Jun 02 15:47:28.205549 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.205533    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:47:28.206835 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.206803    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:47:28.206935 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.206884    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:47:28.333981 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.333941    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9r2\" (UniqueName: \"kubernetes.io/projected/dc37083d-acbb-4bf3-b507-84bf54dd939d-kube-api-access-tm9r2\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.334177 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.333997    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.334177 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.334031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc37083d-acbb-4bf3-b507-84bf54dd939d-tmp-dir\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.334177 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.334101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc37083d-acbb-4bf3-b507-84bf54dd939d-config-volume\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.334336 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.334182    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.334336 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.334217    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4cd\" (UniqueName: \"kubernetes.io/projected/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-kube-api-access-wq4cd\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.435304 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435264    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.435474 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435314    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc37083d-acbb-4bf3-b507-84bf54dd939d-tmp-dir\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.435474 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435359    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc37083d-acbb-4bf3-b507-84bf54dd939d-config-volume\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.435474 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435402    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.435474 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435429    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4cd\" (UniqueName: \"kubernetes.io/projected/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-kube-api-access-wq4cd\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.435474 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.435445    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:28.435730 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435479    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9r2\" (UniqueName: \"kubernetes.io/projected/dc37083d-acbb-4bf3-b507-84bf54dd939d-kube-api-access-tm9r2\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.435730 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.435524    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.935501322 +0000 UTC m=+34.620049249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:28.435730 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.435542    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:28.435730 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.435614    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.935595827 +0000 UTC m=+34.620143763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:28.435932 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.435788    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc37083d-acbb-4bf3-b507-84bf54dd939d-tmp-dir\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.436079 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.436041    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc37083d-acbb-4bf3-b507-84bf54dd939d-config-volume\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.453216 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.453157    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9r2\" (UniqueName: \"kubernetes.io/projected/dc37083d-acbb-4bf3-b507-84bf54dd939d-kube-api-access-tm9r2\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.453345 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.453283    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4cd\" (UniqueName: \"kubernetes.io/projected/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-kube-api-access-wq4cd\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.536311 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.536274    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:28.536478 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.536457    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:28.536555 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.536544    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:00.536524617 +0000 UTC m=+66.221072553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:28.636682 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.636636    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:28.636844 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.636726    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:28.636844 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636811    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:28.636844 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636834    2568 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:28.637007 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636848    2568 projected.go:194] Error preparing data for projected volume kube-api-access-r7l8s for pod openshift-network-diagnostics/network-check-target-69vh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:28.637007 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636873    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:28.637007 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636915    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s podName:2c88dfd2-651a-4626-a690-35e05cc1ba9d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:00.636894319 +0000 UTC m=+66.321442244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r7l8s" (UniqueName: "kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s") pod "network-check-target-69vh8" (UID: "2c88dfd2-651a-4626-a690-35e05cc1ba9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:28.637007 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.636936    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret podName:c59b270b-1208-4a9a-bd68-2ad2507ebf2f nodeName:}" failed. No retries permitted until 2026-06-02 15:48:00.636926229 +0000 UTC m=+66.321474143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret") pod "global-pull-secret-syncer-xfr2t" (UID: "c59b270b-1208-4a9a-bd68-2ad2507ebf2f") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:28.939749 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.939712    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:28.940303 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:28.939780    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:28.940303 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.939866    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:28.940303 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.939925    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:29.939911133 +0000 UTC m=+35.624459048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:28.940303 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.939866    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:28.940303 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:28.940018    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:29.940000622 +0000 UTC m=+35.624548553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:29.889371 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.889191    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:47:29.889555 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.889191    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:47:29.889555 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.889191    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:47:29.901871 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.901847    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m8f5r\""
Jun 02 15:47:29.901871 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.901847    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:47:29.902043 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.901846    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:29.902043 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.901862    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:29.902043 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.901889    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:29.908350 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.908333    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j9fj4\""
Jun 02 15:47:29.950045 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.950021    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:29.950342 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:29.950085    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:29.950342 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:29.950172    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:29.950342 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:29.950232    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:31.95021297 +0000 UTC m=+37.634760898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:29.950342 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:29.950173    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:29.950342 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:29.950294    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:31.950281763 +0000 UTC m=+37.634829694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:30.110732 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:30.110702    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="f92886cc1969b7b697a8ad7dd2371f7cf6a83ad73b8c64a7f911b008f6b460b4" exitCode=0
Jun 02 15:47:30.110732 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:30.110741    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"f92886cc1969b7b697a8ad7dd2371f7cf6a83ad73b8c64a7f911b008f6b460b4"}
Jun 02 15:47:31.115393 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:31.115360    2568 generic.go:358] "Generic (PLEG): container finished" podID="e07b584a-2b02-442f-80f5-e5d7d89f1452" containerID="36a6c8e577a7fecb307db714f5462a5cd1b46ceeb24e8bf9c2a0af4de05abe99" exitCode=0
Jun 02 15:47:31.115767 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:31.115422    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerDied","Data":"36a6c8e577a7fecb307db714f5462a5cd1b46ceeb24e8bf9c2a0af4de05abe99"}
Jun 02 15:47:31.966825 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:31.966791    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:31.966971 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:31.966855    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:31.966971 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:31.966953    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:31.966971 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:31.966956    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:31.967139 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:31.967028    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:35.967005045 +0000 UTC m=+41.651552972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:31.967139 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:31.967069    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:35.96703898 +0000 UTC m=+41.651586893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:32.120336 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.120306    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knr5w" event={"ID":"e07b584a-2b02-442f-80f5-e5d7d89f1452","Type":"ContainerStarted","Data":"7c53212daf31849c7db62c3170ee1f2241409029f74ec59561896ff60e4f2ee7"}
Jun 02 15:47:32.162862 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.162772    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-knr5w" podStartSLOduration=6.348128943 podStartE2EDuration="38.162757733s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:46:57.34858727 +0000 UTC m=+3.033135187" lastFinishedPulling="2026-06-02 15:47:29.163216061 +0000 UTC m=+34.847763977" observedRunningTime="2026-06-02 15:47:32.162421401 +0000 UTC m=+37.846969336" watchObservedRunningTime="2026-06-02 15:47:32.162757733 +0000 UTC m=+37.847305667"
Jun 02 15:47:32.806865 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.806823    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"]
Jun 02 15:47:32.833106 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.833071    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"]
Jun 02 15:47:32.833264 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.833187    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:32.835702 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.835677    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jun 02 15:47:32.835848 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.835764    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jun 02 15:47:32.835923 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.835844    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jun 02 15:47:32.836522 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.836503    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jun 02 15:47:32.836678 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.836587    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-69vnp\""
Jun 02 15:47:32.845500 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.845480    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"]
Jun 02 15:47:32.859084 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.859068    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.863281 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.863262    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jun 02 15:47:32.863389 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.863305    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jun 02 15:47:32.863766 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.863722    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jun 02 15:47:32.868578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.868557    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"]
Jun 02 15:47:32.870207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.870190    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jun 02 15:47:32.972027 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.971991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.972027 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972029    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/da4f30b5-bc20-46df-a1ea-e322369edf1a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.972277 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972082    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hr8\" (UniqueName: \"kubernetes.io/projected/d9516c04-ae15-447b-9924-59fea2308bbb-kube-api-access-n4hr8\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:32.972277 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972106    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9516c04-ae15-447b-9924-59fea2308bbb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:32.972277 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972187    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.972277 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972228    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.972422 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972307    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:32.972422 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:32.972341    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmdv\" (UniqueName: \"kubernetes.io/projected/da4f30b5-bc20-46df-a1ea-e322369edf1a-kube-api-access-vwmdv\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073546 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073467    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073546 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073522    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073546 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073547    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmdv\" (UniqueName: \"kubernetes.io/projected/da4f30b5-bc20-46df-a1ea-e322369edf1a-kube-api-access-vwmdv\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073805 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073587    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073805 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073604    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/da4f30b5-bc20-46df-a1ea-e322369edf1a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.073805 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073637    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hr8\" (UniqueName: \"kubernetes.io/projected/d9516c04-ae15-447b-9924-59fea2308bbb-kube-api-access-n4hr8\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:33.073805 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073665    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9516c04-ae15-447b-9924-59fea2308bbb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:33.073805 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.073698    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.074438 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.074412    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/da4f30b5-bc20-46df-a1ea-e322369edf1a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.077191 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.077169    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.077296 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.077188    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.077344 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.077288    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9516c04-ae15-447b-9924-59fea2308bbb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:33.077344 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.077291    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-hub\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.077598 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.077579    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da4f30b5-bc20-46df-a1ea-e322369edf1a-ca\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.085547 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.085526    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hr8\" (UniqueName: \"kubernetes.io/projected/d9516c04-ae15-447b-9924-59fea2308bbb-kube-api-access-n4hr8\") pod \"managed-serviceaccount-addon-agent-7958847574-gf9h5\" (UID: \"d9516c04-ae15-447b-9924-59fea2308bbb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:33.085659 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.085638    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmdv\" (UniqueName: \"kubernetes.io/projected/da4f30b5-bc20-46df-a1ea-e322369edf1a-kube-api-access-vwmdv\") pod \"cluster-proxy-proxy-agent-844596bc8d-5m2s7\" (UID: \"da4f30b5-bc20-46df-a1ea-e322369edf1a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.153125 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.153102    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"
Jun 02 15:47:33.167755 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.167733    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:47:33.322465 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.322251    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5"]
Jun 02 15:47:33.332523 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:47:33.332496    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9516c04_ae15_447b_9924_59fea2308bbb.slice/crio-b66c974f4c986f280183d0fead969d16205e5ef5d5842f1f16f052da7bcc5d97 WatchSource:0}: Error finding container b66c974f4c986f280183d0fead969d16205e5ef5d5842f1f16f052da7bcc5d97: Status 404 returned error can't find the container with id b66c974f4c986f280183d0fead969d16205e5ef5d5842f1f16f052da7bcc5d97
Jun 02 15:47:33.333809 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:33.333787    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"]
Jun 02 15:47:33.337867 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:47:33.337800    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4f30b5_bc20_46df_a1ea_e322369edf1a.slice/crio-ebf215e5e0647c4e2d0d2b7cb681de222a49c2b38ba2a726e66db099bb3e9463 WatchSource:0}: Error finding container ebf215e5e0647c4e2d0d2b7cb681de222a49c2b38ba2a726e66db099bb3e9463: Status 404 returned error can't find the container with id ebf215e5e0647c4e2d0d2b7cb681de222a49c2b38ba2a726e66db099bb3e9463
Jun 02 15:47:34.126433 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:34.126396    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerStarted","Data":"ebf215e5e0647c4e2d0d2b7cb681de222a49c2b38ba2a726e66db099bb3e9463"}
Jun 02 15:47:34.127919 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:34.127886    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5" event={"ID":"d9516c04-ae15-447b-9924-59fea2308bbb","Type":"ContainerStarted","Data":"b66c974f4c986f280183d0fead969d16205e5ef5d5842f1f16f052da7bcc5d97"}
Jun 02 15:47:35.997257 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:35.997210    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:35.997703 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:35.997294    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:35.997703 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:35.997352    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:35.997703 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:35.997398    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:35.997703 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:35.997431    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:43.997404161 +0000 UTC m=+49.681952094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:35.997703 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:35.997463    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:43.997440749 +0000 UTC m=+49.681988667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:37.134551 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:37.134517    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5" event={"ID":"d9516c04-ae15-447b-9924-59fea2308bbb","Type":"ContainerStarted","Data":"7e6529223d595813f3c24713e58465e2bb1077f015bc8ec5a85deac52001e242"}
Jun 02 15:47:37.158589 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:37.158539    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5" podStartSLOduration=1.817477773 podStartE2EDuration="5.1584824s" podCreationTimestamp="2026-06-02 15:47:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:33.334866026 +0000 UTC m=+39.019413945" lastFinishedPulling="2026-06-02 15:47:36.675870641 +0000 UTC m=+42.360418572" observedRunningTime="2026-06-02 15:47:37.158371211 +0000 UTC m=+42.842919146" watchObservedRunningTime="2026-06-02 15:47:37.1584824 +0000 UTC m=+42.843030334"
Jun 02 15:47:38.139511 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:38.139475    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerStarted","Data":"a397e67a06bedde24ed736b37d47fdfd4e1b14dc321a6978c912edbf344ec2ed"}
Jun 02 15:47:44.055873 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:44.055835    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:47:44.056306 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:44.055899    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:47:44.056306 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:44.055987    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:44.056306 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:44.056032    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:00.056019053 +0000 UTC m=+65.740566966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:44.056306 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:44.056104    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:44.056306 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:47:44.056159    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:00.056146276 +0000 UTC m=+65.740694193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:47:45.154031 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:45.153996    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerStarted","Data":"6d8d0121e936c72496e7bb0c3124dad6b696ae78546faaef5405f5a82b4251d9"}
Jun 02 15:47:45.154031 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:45.154032    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerStarted","Data":"506b8713e40ebc3683f8b813b872452c11ff438922ae904d116f095455da4c01"}
Jun 02 15:47:45.181723 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:45.181668    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" podStartSLOduration=2.248153863 podStartE2EDuration="13.181655577s" podCreationTimestamp="2026-06-02 15:47:32 +0000 UTC" firstStartedPulling="2026-06-02 15:47:33.339506092 +0000 UTC m=+39.024054004" lastFinishedPulling="2026-06-02 15:47:44.273007805 +0000 UTC m=+49.957555718" observedRunningTime="2026-06-02 15:47:45.180480239 +0000 UTC m=+50.865028173" watchObservedRunningTime="2026-06-02 15:47:45.181655577 +0000 UTC m=+50.866203512"
Jun 02 15:47:54.107774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:47:54.107746    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79m7"
Jun 02 15:48:00.071379 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.071338    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:48:00.071947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.071395    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:48:00.071947 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.071499    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:48:00.071947 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.071508    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:48:00.071947 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.071555    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:32.071538814 +0000 UTC m=+97.756086730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:48:00.071947 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.071569    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:32.071563057 +0000 UTC m=+97.756110970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:48:00.576152 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.576114    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:48:00.578871 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.578850    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:48:00.587043 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.587025    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:48:00.587133 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:00.587091    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:04.587077482 +0000 UTC m=+130.271625395 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : secret "metrics-daemon-secret" not found
Jun 02 15:48:00.676877 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.676837    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:48:00.677033 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.676906    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:48:00.679679 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.679659    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:48:00.680179 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.680161    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:48:00.689613 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.689592    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:48:00.690202 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.690183    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c59b270b-1208-4a9a-bd68-2ad2507ebf2f-original-pull-secret\") pod \"global-pull-secret-syncer-xfr2t\" (UID: \"c59b270b-1208-4a9a-bd68-2ad2507ebf2f\") " pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:48:00.699707 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.699688    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7l8s\" (UniqueName: \"kubernetes.io/projected/2c88dfd2-651a-4626-a690-35e05cc1ba9d-kube-api-access-r7l8s\") pod \"network-check-target-69vh8\" (UID: \"2c88dfd2-651a-4626-a690-35e05cc1ba9d\") " pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:48:00.801418 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.801383    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m8f5r\""
Jun 02 15:48:00.804904 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.804885    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xfr2t"
Jun 02 15:48:00.808606 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.808590    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:48:00.951461 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.951431    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69vh8"]
Jun 02 15:48:00.954249 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:48:00.954224    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c88dfd2_651a_4626_a690_35e05cc1ba9d.slice/crio-c97830bc616711479f9a798cef18b1237910d20d4780db8675cc8e7eae9e167d WatchSource:0}: Error finding container c97830bc616711479f9a798cef18b1237910d20d4780db8675cc8e7eae9e167d: Status 404 returned error can't find the container with id c97830bc616711479f9a798cef18b1237910d20d4780db8675cc8e7eae9e167d
Jun 02 15:48:00.964446 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:00.964423    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xfr2t"]
Jun 02 15:48:00.967904 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:48:00.967883    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59b270b_1208_4a9a_bd68_2ad2507ebf2f.slice/crio-02eec76b1896331d41a4b380b4624bc241af22edb04d363f8748f1fd254ce815 WatchSource:0}: Error finding container 02eec76b1896331d41a4b380b4624bc241af22edb04d363f8748f1fd254ce815: Status 404 returned error can't find the container with id 02eec76b1896331d41a4b380b4624bc241af22edb04d363f8748f1fd254ce815
Jun 02 15:48:01.185616 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:01.185584    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69vh8" event={"ID":"2c88dfd2-651a-4626-a690-35e05cc1ba9d","Type":"ContainerStarted","Data":"c97830bc616711479f9a798cef18b1237910d20d4780db8675cc8e7eae9e167d"}
Jun 02 15:48:01.186601 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:01.186575    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfr2t" event={"ID":"c59b270b-1208-4a9a-bd68-2ad2507ebf2f","Type":"ContainerStarted","Data":"02eec76b1896331d41a4b380b4624bc241af22edb04d363f8748f1fd254ce815"}
Jun 02 15:48:06.197668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:06.197629    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69vh8" event={"ID":"2c88dfd2-651a-4626-a690-35e05cc1ba9d","Type":"ContainerStarted","Data":"ba855a0f85b71d776d1df4286f5bedac52f5edcd1ca205984a4c2a5ce58f9ac4"}
Jun 02 15:48:06.198145 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:06.197739    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:48:06.201075 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:06.201031    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xfr2t" event={"ID":"c59b270b-1208-4a9a-bd68-2ad2507ebf2f","Type":"ContainerStarted","Data":"48c985cb2b3470e60ed14a06f64d484282aeb66d347b02977ddd127a081fc146"}
Jun 02 15:48:06.217398 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:06.217351    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-69vh8" podStartSLOduration=67.02951506 podStartE2EDuration="1m11.217338506s" podCreationTimestamp="2026-06-02 15:46:55 +0000 UTC" firstStartedPulling="2026-06-02 15:48:00.955997982 +0000 UTC m=+66.640545895" lastFinishedPulling="2026-06-02 15:48:05.143821422 +0000 UTC m=+70.828369341" observedRunningTime="2026-06-02 15:48:06.21671978 +0000 UTC m=+71.901267715" watchObservedRunningTime="2026-06-02 15:48:06.217338506 +0000 UTC m=+71.901886441"
Jun 02 15:48:06.232735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:06.232688    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xfr2t" podStartSLOduration=66.056358937 podStartE2EDuration="1m10.232673508s" podCreationTimestamp="2026-06-02 15:46:56 +0000 UTC" firstStartedPulling="2026-06-02 15:48:00.969573124 +0000 UTC m=+66.654121037" lastFinishedPulling="2026-06-02 15:48:05.145887693 +0000 UTC m=+70.830435608" observedRunningTime="2026-06-02 15:48:06.232477819 +0000 UTC m=+71.917025755" watchObservedRunningTime="2026-06-02 15:48:06.232673508 +0000 UTC m=+71.917221444"
Jun 02 15:48:32.101189 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:32.101151    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:48:32.101594 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:32.101226    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:48:32.101594 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:32.101307    2568 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:48:32.101594 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:32.101368    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert podName:e9c5a910-06ea-4bd4-9afb-a5c61d74f572 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:36.101354147 +0000 UTC m=+161.785902064 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert") pod "ingress-canary-b98jg" (UID: "e9c5a910-06ea-4bd4-9afb-a5c61d74f572") : secret "canary-serving-cert" not found
Jun 02 15:48:32.101594 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:32.101311    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:48:32.101594 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:48:32.101442    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls podName:dc37083d-acbb-4bf3-b507-84bf54dd939d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:36.101431369 +0000 UTC m=+161.785979296 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls") pod "dns-default-7zqcd" (UID: "dc37083d-acbb-4bf3-b507-84bf54dd939d") : secret "dns-default-metrics-tls" not found
Jun 02 15:48:37.206384 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:37.206350    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-69vh8"
Jun 02 15:48:57.954716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:57.954687    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gltw9_099e71f1-a837-487b-a2e1-70e9e992800a/dns-node-resolver/0.log"
Jun 02 15:48:58.755995 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:48:58.755970    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tqbr4_72ba1511-d074-4216-ad97-1459127915d0/node-ca/0.log"
Jun 02 15:49:04.625929 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:04.625888    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:49:04.626435 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:04.626030    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:49:04.626435 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:04.626122    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs podName:c0186ca7-bafe-4ba4-82bf-0257fbff4133 nodeName:}" failed. No retries permitted until 2026-06-02 15:51:06.626107538 +0000 UTC m=+252.310655451 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs") pod "network-metrics-daemon-xtvfm" (UID: "c0186ca7-bafe-4ba4-82bf-0257fbff4133") : secret "metrics-daemon-secret" not found
Jun 02 15:49:18.688558 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.688520    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z9cxq"]
Jun 02 15:49:18.691340 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.691325    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.697999 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.697151    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:49:18.697999 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.697724    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:49:18.698356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.698339    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-589bn\""
Jun 02 15:49:18.698421 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.698401    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:49:18.699097 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.699077    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:49:18.709785 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.709763    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9cxq"]
Jun 02 15:49:18.828166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.828131    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4968d9d2-480a-406d-85c6-ae2468b7e837-crio-socket\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.828166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.828169    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4hb\" (UniqueName: \"kubernetes.io/projected/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-api-access-db4hb\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.828364 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.828219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4968d9d2-480a-406d-85c6-ae2468b7e837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.828364 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.828317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4968d9d2-480a-406d-85c6-ae2468b7e837-data-volume\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.828364 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.828344    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929203 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929169    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4968d9d2-480a-406d-85c6-ae2468b7e837-crio-socket\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929203 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929204    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-db4hb\" (UniqueName: \"kubernetes.io/projected/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-api-access-db4hb\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929442 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929235    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4968d9d2-480a-406d-85c6-ae2468b7e837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929442 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929252    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4968d9d2-480a-406d-85c6-ae2468b7e837-data-volume\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929442 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929268    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929442 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929294    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4968d9d2-480a-406d-85c6-ae2468b7e837-crio-socket\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929634 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929616    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4968d9d2-480a-406d-85c6-ae2468b7e837-data-volume\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.929767 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.929752    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.931609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.931586    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4968d9d2-480a-406d-85c6-ae2468b7e837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:18.946412 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:18.946358    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4hb\" (UniqueName: \"kubernetes.io/projected/4968d9d2-480a-406d-85c6-ae2468b7e837-kube-api-access-db4hb\") pod \"insights-runtime-extractor-z9cxq\" (UID: \"4968d9d2-480a-406d-85c6-ae2468b7e837\") " pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:19.000093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:19.000062    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z9cxq"
Jun 02 15:49:19.119759 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:19.119731    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z9cxq"]
Jun 02 15:49:19.123351 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:19.123325    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4968d9d2_480a_406d_85c6_ae2468b7e837.slice/crio-8462fc1c561ad3664d359e14dc226cf4c4e1c7b8e82bbb6b429abb1f57d384ec WatchSource:0}: Error finding container 8462fc1c561ad3664d359e14dc226cf4c4e1c7b8e82bbb6b429abb1f57d384ec: Status 404 returned error can't find the container with id 8462fc1c561ad3664d359e14dc226cf4c4e1c7b8e82bbb6b429abb1f57d384ec
Jun 02 15:49:19.371283 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:19.371201    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9cxq" event={"ID":"4968d9d2-480a-406d-85c6-ae2468b7e837","Type":"ContainerStarted","Data":"59cbb97cbbe3e93e68b2059b31a13c0e011fbbd023182d30b7439b99ffff31a6"}
Jun 02 15:49:19.371283 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:19.371237    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9cxq" event={"ID":"4968d9d2-480a-406d-85c6-ae2468b7e837","Type":"ContainerStarted","Data":"8462fc1c561ad3664d359e14dc226cf4c4e1c7b8e82bbb6b429abb1f57d384ec"}
Jun 02 15:49:20.380436 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:20.380400    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9cxq" event={"ID":"4968d9d2-480a-406d-85c6-ae2468b7e837","Type":"ContainerStarted","Data":"a3021908b32da28a8570c2af110ffba49a475be800f08b7ca31e6d66cd3fb93c"}
Jun 02 15:49:21.384720 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:21.384686    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z9cxq" event={"ID":"4968d9d2-480a-406d-85c6-ae2468b7e837","Type":"ContainerStarted","Data":"6a34c3ca9e0d65802447f8dc1aa1f7581436990d54b3ec72bff4a34932fd8d7d"}
Jun 02 15:49:21.402606 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:21.402557    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z9cxq" podStartSLOduration=1.469243653 podStartE2EDuration="3.402541293s" podCreationTimestamp="2026-06-02 15:49:18 +0000 UTC" firstStartedPulling="2026-06-02 15:49:19.175249701 +0000 UTC m=+144.859797615" lastFinishedPulling="2026-06-02 15:49:21.108547335 +0000 UTC m=+146.793095255" observedRunningTime="2026-06-02 15:49:21.402028314 +0000 UTC m=+147.086576248" watchObservedRunningTime="2026-06-02 15:49:21.402541293 +0000 UTC m=+147.087089218"
Jun 02 15:49:30.654046 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.654013    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:49:30.656990 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.656973    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.660208 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.660190    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 02 15:49:30.660554 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.660534    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 02 15:49:30.660768 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.660745    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 02 15:49:30.660768 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.660761    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 02 15:49:30.661162 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.661146    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 02 15:49:30.661447 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.661426    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 02 15:49:30.661551 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.661462    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-928l8\""
Jun 02 15:49:30.661551 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.661469    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 02 15:49:30.665901 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.665882    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jun 02 15:49:30.668367 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.668346    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:49:30.715263 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715229    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715263 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715262    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715332    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715358    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715387    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.715435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.715410    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6s6\" (UniqueName: \"kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.815901 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.815864    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.815901 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.815904    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816129 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.815931    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816129 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816038    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816129 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816092    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816129 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816125    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816323 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816142    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6s6\" (UniqueName: \"kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816771 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816748    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816860 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816810    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816937 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816915    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.816969 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.816951    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.818547 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.818529    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.818595 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.818537    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.826740 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.826712    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6s6\" (UniqueName: \"kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6\") pod \"console-59666dd688-6g566\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") " pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:30.965826 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:30.965733    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:31.090142 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:31.090105    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:49:31.093236 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:31.093207    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2d9c0b_d147_443d_aaee_56cb893dc2a4.slice/crio-38eed7ddd87f64d7903a98736eebd8d6255d4d6e04d4b87e07729650b73be5cd WatchSource:0}: Error finding container 38eed7ddd87f64d7903a98736eebd8d6255d4d6e04d4b87e07729650b73be5cd: Status 404 returned error can't find the container with id 38eed7ddd87f64d7903a98736eebd8d6255d4d6e04d4b87e07729650b73be5cd
Jun 02 15:49:31.208854 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:31.208810    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7zqcd" podUID="dc37083d-acbb-4bf3-b507-84bf54dd939d"
Jun 02 15:49:31.216167 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:31.216100    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-b98jg" podUID="e9c5a910-06ea-4bd4-9afb-a5c61d74f572"
Jun 02 15:49:31.412232 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:31.412203    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:49:31.412414 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:31.412195    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59666dd688-6g566" event={"ID":"ae2d9c0b-d147-443d-aaee-56cb893dc2a4","Type":"ContainerStarted","Data":"38eed7ddd87f64d7903a98736eebd8d6255d4d6e04d4b87e07729650b73be5cd"}
Jun 02 15:49:31.412414 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:31.412324    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:32.050010 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.049974    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4vswl"]
Jun 02 15:49:32.052945 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.052928    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.055711 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.055684    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:49:32.055851 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.055684    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:49:32.056382 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.056357    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:49:32.056382 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.056370    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wjpl\""
Jun 02 15:49:32.056752 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.056380    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:49:32.056752 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.056506    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:49:32.056752 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.056511    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:49:32.068707 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.068678    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"]
Jun 02 15:49:32.071807 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.071792    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.074942 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.074918    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:49:32.075037 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.074952    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qvlm5\""
Jun 02 15:49:32.075037 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.074956    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 02 15:49:32.075214 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.075199    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 02 15:49:32.085918 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.085891    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"]
Jun 02 15:49:32.125515 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125480    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-wtmp\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.125515 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5z4\" (UniqueName: \"kubernetes.io/projected/8a67760f-b952-4ee6-b439-b13ce2461ca6-kube-api-access-sp5z4\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.125705 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8h6\" (UniqueName: \"kubernetes.io/projected/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-api-access-wf8h6\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.125705 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125644    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-sys\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.125705 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125700    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125775    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-textfile\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125797    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69f3fc2c-9f29-43df-9567-93e2e8eaec22-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125820    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.125849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125848    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-tls\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.126122 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125908    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.126122 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125935    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-metrics-client-ca\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.126122 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125952    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-root\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.126122 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.125969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227233 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227140    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227233 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227192    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227233 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227223    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227252    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-textfile\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227275    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69f3fc2c-9f29-43df-9567-93e2e8eaec22-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227300    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227325    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-tls\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227377    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:32.227389    2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227427    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-metrics-client-ca\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227455    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-root\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227494 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:32.227476    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls podName:69f3fc2c-9f29-43df-9567-93e2e8eaec22 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:32.72745417 +0000 UTC m=+158.412002087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls") pod "kube-state-metrics-8469c486bb-6zlfb" (UID: "69f3fc2c-9f29-43df-9567-93e2e8eaec22") : secret "kube-state-metrics-tls" not found
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227507    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-root\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227514    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227577    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-wtmp\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227601    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5z4\" (UniqueName: \"kubernetes.io/projected/8a67760f-b952-4ee6-b439-b13ce2461ca6-kube-api-access-sp5z4\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227625    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8h6\" (UniqueName: \"kubernetes.io/projected/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-api-access-wf8h6\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227653    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-sys\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227713    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-textfile\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227731    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-sys\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.227922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.227849    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-wtmp\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.228378 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.228024    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.228378 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.228068    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.228378 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.228194    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a67760f-b952-4ee6-b439-b13ce2461ca6-metrics-client-ca\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.228378 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.228322    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/69f3fc2c-9f29-43df-9567-93e2e8eaec22-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.228740 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.228686    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.230602 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.230580    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.230716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.230617    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.230786 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.230707    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a67760f-b952-4ee6-b439-b13ce2461ca6-node-exporter-tls\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.240223 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.240202    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5z4\" (UniqueName: \"kubernetes.io/projected/8a67760f-b952-4ee6-b439-b13ce2461ca6-kube-api-access-sp5z4\") pod \"node-exporter-4vswl\" (UID: \"8a67760f-b952-4ee6-b439-b13ce2461ca6\") " pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.240721 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.240701    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8h6\" (UniqueName: \"kubernetes.io/projected/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-api-access-wf8h6\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.363001 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.362964    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4vswl"
Jun 02 15:49:32.373012 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:32.372974    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a67760f_b952_4ee6_b439_b13ce2461ca6.slice/crio-a1d67662dc98bd2a18a8e105d5bf7c3ce74c452ecaf2dde4b8e774fe2ba5177f WatchSource:0}: Error finding container a1d67662dc98bd2a18a8e105d5bf7c3ce74c452ecaf2dde4b8e774fe2ba5177f: Status 404 returned error can't find the container with id a1d67662dc98bd2a18a8e105d5bf7c3ce74c452ecaf2dde4b8e774fe2ba5177f
Jun 02 15:49:32.415002 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.414964    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vswl" event={"ID":"8a67760f-b952-4ee6-b439-b13ce2461ca6","Type":"ContainerStarted","Data":"a1d67662dc98bd2a18a8e105d5bf7c3ce74c452ecaf2dde4b8e774fe2ba5177f"}
Jun 02 15:49:32.732540 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.732495    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.735448 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.735406    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69f3fc2c-9f29-43df-9567-93e2e8eaec22-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-6zlfb\" (UID: \"69f3fc2c-9f29-43df-9567-93e2e8eaec22\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:32.908845 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:49:32.908801    2568 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xtvfm" podUID="c0186ca7-bafe-4ba4-82bf-0257fbff4133"
Jun 02 15:49:32.980448 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:32.980412    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"
Jun 02 15:49:34.351451 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.351397    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb"]
Jun 02 15:49:34.355843 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:34.355812    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f3fc2c_9f29_43df_9567_93e2e8eaec22.slice/crio-8492a56cc07ea063d4adedaca5eec6ba404d722f82dd16cc6416ee949f9a0428 WatchSource:0}: Error finding container 8492a56cc07ea063d4adedaca5eec6ba404d722f82dd16cc6416ee949f9a0428: Status 404 returned error can't find the container with id 8492a56cc07ea063d4adedaca5eec6ba404d722f82dd16cc6416ee949f9a0428
Jun 02 15:49:34.421858 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.421821    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59666dd688-6g566" event={"ID":"ae2d9c0b-d147-443d-aaee-56cb893dc2a4","Type":"ContainerStarted","Data":"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"}
Jun 02 15:49:34.422943 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.422916    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb" event={"ID":"69f3fc2c-9f29-43df-9567-93e2e8eaec22","Type":"ContainerStarted","Data":"8492a56cc07ea063d4adedaca5eec6ba404d722f82dd16cc6416ee949f9a0428"}
Jun 02 15:49:34.424227 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.424200    2568 generic.go:358] "Generic (PLEG): container finished" podID="8a67760f-b952-4ee6-b439-b13ce2461ca6" containerID="5e6ff42322468f10b02da6b6b9759bc8529d89b56e2da1a2e661f2903a5308d9" exitCode=0
Jun 02 15:49:34.424317 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.424241    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vswl" event={"ID":"8a67760f-b952-4ee6-b439-b13ce2461ca6","Type":"ContainerDied","Data":"5e6ff42322468f10b02da6b6b9759bc8529d89b56e2da1a2e661f2903a5308d9"}
Jun 02 15:49:34.441418 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:34.441371    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-59666dd688-6g566" podStartSLOduration=1.306221232 podStartE2EDuration="4.441354518s" podCreationTimestamp="2026-06-02 15:49:30 +0000 UTC" firstStartedPulling="2026-06-02 15:49:31.095019107 +0000 UTC m=+156.779567020" lastFinishedPulling="2026-06-02 15:49:34.23015239 +0000 UTC m=+159.914700306" observedRunningTime="2026-06-02 15:49:34.441289742 +0000 UTC m=+160.125837679" watchObservedRunningTime="2026-06-02 15:49:34.441354518 +0000 UTC m=+160.125902506"
Jun 02 15:49:35.429155 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:35.429099    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vswl" event={"ID":"8a67760f-b952-4ee6-b439-b13ce2461ca6","Type":"ContainerStarted","Data":"e2b67da592a9e9e8c780d8be5addca98f8d0e2bcd6229579e26027bf4a3061d2"}
Jun 02 15:49:35.429155 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:35.429153    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vswl" event={"ID":"8a67760f-b952-4ee6-b439-b13ce2461ca6","Type":"ContainerStarted","Data":"bfd86b307a11c102f4ab43b0eaf58f1760d359ab6c3b0c4d746388b8261a0035"}
Jun 02 15:49:35.447514 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:35.447464    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4vswl" podStartSLOduration=1.6023950139999998 podStartE2EDuration="3.447448977s" podCreationTimestamp="2026-06-02 15:49:32 +0000 UTC" firstStartedPulling="2026-06-02 15:49:32.375713298 +0000 UTC m=+158.060261214" lastFinishedPulling="2026-06-02 15:49:34.22076726 +0000 UTC m=+159.905315177" observedRunningTime="2026-06-02 15:49:35.446600418 +0000 UTC m=+161.131148353" watchObservedRunningTime="2026-06-02 15:49:35.447448977 +0000 UTC m=+161.131996911"
Jun 02 15:49:36.163903 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.163870    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:49:36.164126 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.163926    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:36.166294 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.166263    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc37083d-acbb-4bf3-b507-84bf54dd939d-metrics-tls\") pod \"dns-default-7zqcd\" (UID: \"dc37083d-acbb-4bf3-b507-84bf54dd939d\") " pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:36.166492 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.166446    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9c5a910-06ea-4bd4-9afb-a5c61d74f572-cert\") pod \"ingress-canary-b98jg\" (UID: \"e9c5a910-06ea-4bd4-9afb-a5c61d74f572\") " pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:49:36.215718 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.215687    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25zcr\""
Jun 02 15:49:36.215891 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.215840    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gg4df\""
Jun 02 15:49:36.223575 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.223555    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:36.223695 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.223633    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b98jg"
Jun 02 15:49:36.355862 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.355829    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zqcd"]
Jun 02 15:49:36.358501 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:36.358474    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc37083d_acbb_4bf3_b507_84bf54dd939d.slice/crio-494db0c999135a9434fb261caed4dc15862dce224e7715d82badf9deb396c876 WatchSource:0}: Error finding container 494db0c999135a9434fb261caed4dc15862dce224e7715d82badf9deb396c876: Status 404 returned error can't find the container with id 494db0c999135a9434fb261caed4dc15862dce224e7715d82badf9deb396c876
Jun 02 15:49:36.370530 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.370509    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b98jg"]
Jun 02 15:49:36.372725 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:36.372701    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c5a910_06ea_4bd4_9afb_a5c61d74f572.slice/crio-5ed36183ee90e56159485b8f27514046ef49c67f3a151838b7b4654d909a0b0d WatchSource:0}: Error finding container 5ed36183ee90e56159485b8f27514046ef49c67f3a151838b7b4654d909a0b0d: Status 404 returned error can't find the container with id 5ed36183ee90e56159485b8f27514046ef49c67f3a151838b7b4654d909a0b0d
Jun 02 15:49:36.432960 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.432874    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb" event={"ID":"69f3fc2c-9f29-43df-9567-93e2e8eaec22","Type":"ContainerStarted","Data":"64a2a9a13be3ebbb9098b5f5b07a929d2e77eef35b73546b7fd65ee8f651111d"}
Jun 02 15:49:36.432960 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.432919    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb" event={"ID":"69f3fc2c-9f29-43df-9567-93e2e8eaec22","Type":"ContainerStarted","Data":"1420251d1786af1260e50f4a2e8fd50d8148c7dcde3b5f1206a2745e7c2e2c71"}
Jun 02 15:49:36.432960 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.432936    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb" event={"ID":"69f3fc2c-9f29-43df-9567-93e2e8eaec22","Type":"ContainerStarted","Data":"2a64a670a292f833fe7527489103716ab04c1ef1598034de185f63c4ba26096c"}
Jun 02 15:49:36.433978 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.433953    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zqcd" event={"ID":"dc37083d-acbb-4bf3-b507-84bf54dd939d","Type":"ContainerStarted","Data":"494db0c999135a9434fb261caed4dc15862dce224e7715d82badf9deb396c876"}
Jun 02 15:49:36.434766 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.434747    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b98jg" event={"ID":"e9c5a910-06ea-4bd4-9afb-a5c61d74f572","Type":"ContainerStarted","Data":"5ed36183ee90e56159485b8f27514046ef49c67f3a151838b7b4654d909a0b0d"}
Jun 02 15:49:36.451432 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.451392    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-8469c486bb-6zlfb" podStartSLOduration=3.031204563 podStartE2EDuration="4.451380778s" podCreationTimestamp="2026-06-02 15:49:32 +0000 UTC" firstStartedPulling="2026-06-02 15:49:34.358041056 +0000 UTC m=+160.042588969" lastFinishedPulling="2026-06-02 15:49:35.778217271 +0000 UTC m=+161.462765184" observedRunningTime="2026-06-02 15:49:36.450432607 +0000 UTC m=+162.134980541" watchObservedRunningTime="2026-06-02 15:49:36.451380778 +0000 UTC m=+162.135928714"
Jun 02 15:49:36.477565 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.477534    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8645bc875b-nbzvh"]
Jun 02 15:49:36.482668 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.482648    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.485330 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485311    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-frlualhc01k0\""
Jun 02 15:49:36.485440 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485385    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jun 02 15:49:36.485508 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485467    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jun 02 15:49:36.485894 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485875    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jun 02 15:49:36.485997 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485908    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qhxb5\""
Jun 02 15:49:36.485997 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.485973    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jun 02 15:49:36.490463 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.490442    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8645bc875b-nbzvh"]
Jun 02 15:49:36.568171 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-metrics-server-audit-profiles\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568171 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568175    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-client-certs\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568362 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568196    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568362 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w4x\" (UniqueName: \"kubernetes.io/projected/10097207-4680-4999-9bcd-776f85ae34af-kube-api-access-g5w4x\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568362 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568291    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-tls\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568362 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568335    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/10097207-4680-4999-9bcd-776f85ae34af-audit-log\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.568485 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.568410    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-client-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669512 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669466    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-metrics-server-audit-profiles\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669512 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669519    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-client-certs\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669745 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669540    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669745 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669572    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w4x\" (UniqueName: \"kubernetes.io/projected/10097207-4680-4999-9bcd-776f85ae34af-kube-api-access-g5w4x\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669745 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669596    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-tls\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669745 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669649    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/10097207-4680-4999-9bcd-776f85ae34af-audit-log\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.669947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.669780    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-client-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.670084 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.670042    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/10097207-4680-4999-9bcd-776f85ae34af-audit-log\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.670310 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.670285    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.670656 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.670631    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/10097207-4680-4999-9bcd-776f85ae34af-metrics-server-audit-profiles\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.671947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.671927    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-client-certs\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.672121 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.672104    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-client-ca-bundle\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.672167 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.672146    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/10097207-4680-4999-9bcd-776f85ae34af-secret-metrics-server-tls\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.678325 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.678307    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w4x\" (UniqueName: \"kubernetes.io/projected/10097207-4680-4999-9bcd-776f85ae34af-kube-api-access-g5w4x\") pod \"metrics-server-8645bc875b-nbzvh\" (UID: \"10097207-4680-4999-9bcd-776f85ae34af\") " pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.793029 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.792950    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:36.919207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:36.919177    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8645bc875b-nbzvh"]
Jun 02 15:49:36.922128 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:49:36.922101    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10097207_4680_4999_9bcd_776f85ae34af.slice/crio-b9020f39f98a2528ff811dc11a368b3e10dc539be2ec1db6743e85e0cc13f977 WatchSource:0}: Error finding container b9020f39f98a2528ff811dc11a368b3e10dc539be2ec1db6743e85e0cc13f977: Status 404 returned error can't find the container with id b9020f39f98a2528ff811dc11a368b3e10dc539be2ec1db6743e85e0cc13f977
Jun 02 15:49:37.440122 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:37.440082    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh" event={"ID":"10097207-4680-4999-9bcd-776f85ae34af","Type":"ContainerStarted","Data":"b9020f39f98a2528ff811dc11a368b3e10dc539be2ec1db6743e85e0cc13f977"}
Jun 02 15:49:37.441528 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:37.441490    2568 generic.go:358] "Generic (PLEG): container finished" podID="d9516c04-ae15-447b-9924-59fea2308bbb" containerID="7e6529223d595813f3c24713e58465e2bb1077f015bc8ec5a85deac52001e242" exitCode=255
Jun 02 15:49:37.441730 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:37.441692    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5" event={"ID":"d9516c04-ae15-447b-9924-59fea2308bbb","Type":"ContainerDied","Data":"7e6529223d595813f3c24713e58465e2bb1077f015bc8ec5a85deac52001e242"}
Jun 02 15:49:37.442178 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:37.442160    2568 scope.go:117] "RemoveContainer" containerID="7e6529223d595813f3c24713e58465e2bb1077f015bc8ec5a85deac52001e242"
Jun 02 15:49:38.505696 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.505657    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:49:38.508861 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.508841    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.519778 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.519749    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:49:38.585985 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.585949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.585985 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.585986    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.586239 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.586017    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.586239 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.586111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.586239 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.586142    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.586351 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.586244    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njn4q\" (UniqueName: \"kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.586351 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.586281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687436 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687355    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-njn4q\" (UniqueName: \"kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687436 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687403    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687445    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687473    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687505    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687543    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.687663 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.687567    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.688437 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.688361    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.688437 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.688363    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.688605 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.688463    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.688605 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.688574    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.690273 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.690252    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.690373 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.690355    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.706379 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.706352    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njn4q\" (UniqueName: \"kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q\") pod \"console-77c975d575-vh4b6\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") " pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:38.820034 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:38.820008    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:39.019967 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.019934    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:49:39.448760 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.448722    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh" event={"ID":"10097207-4680-4999-9bcd-776f85ae34af","Type":"ContainerStarted","Data":"4ace5899d38524c326e61a8eefadfd166ece82368b4606542fe8f4d30a7a33a5"}
Jun 02 15:49:39.450130 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.450105    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c975d575-vh4b6" event={"ID":"985837be-0bc6-4b60-8951-629511d04052","Type":"ContainerStarted","Data":"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"}
Jun 02 15:49:39.450262 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.450133    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c975d575-vh4b6" event={"ID":"985837be-0bc6-4b60-8951-629511d04052","Type":"ContainerStarted","Data":"f832d0924b1dfbc4b9e430665c52f78056b61cdc2b04121ea287635a80c76248"}
Jun 02 15:49:39.451632 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.451602    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7958847574-gf9h5" event={"ID":"d9516c04-ae15-447b-9924-59fea2308bbb","Type":"ContainerStarted","Data":"8d5143f82fc7a59459e68cfbed4f2ad5b9693de6b680d01a414ddac5969bfba5"}
Jun 02 15:49:39.453156 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.453137    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zqcd" event={"ID":"dc37083d-acbb-4bf3-b507-84bf54dd939d","Type":"ContainerStarted","Data":"2ccc3498807920d2779ad8d46492ea105e9c8671d5cfc09d1367988f93cb616e"}
Jun 02 15:49:39.453242 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.453160    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zqcd" event={"ID":"dc37083d-acbb-4bf3-b507-84bf54dd939d","Type":"ContainerStarted","Data":"1ab227b722dd69fec7cb8ebe04ee508a1fae33a0063371deee3103d082bbd09a"}
Jun 02 15:49:39.453306 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.453273    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:39.454392 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.454367    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b98jg" event={"ID":"e9c5a910-06ea-4bd4-9afb-a5c61d74f572","Type":"ContainerStarted","Data":"6f4699f4c218091e2a222e07133d638c3bda621eac94373eab12ba86d4737a1b"}
Jun 02 15:49:39.468237 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.468196    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh" podStartSLOduration=1.5489260329999999 podStartE2EDuration="3.46818675s" podCreationTimestamp="2026-06-02 15:49:36 +0000 UTC" firstStartedPulling="2026-06-02 15:49:36.923892338 +0000 UTC m=+162.608440251" lastFinishedPulling="2026-06-02 15:49:38.843153044 +0000 UTC m=+164.527700968" observedRunningTime="2026-06-02 15:49:39.467783331 +0000 UTC m=+165.152331278" watchObservedRunningTime="2026-06-02 15:49:39.46818675 +0000 UTC m=+165.152734685"
Jun 02 15:49:39.486234 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.486195    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b98jg" podStartSLOduration=129.017517356 podStartE2EDuration="2m11.486181681s" podCreationTimestamp="2026-06-02 15:47:28 +0000 UTC" firstStartedPulling="2026-06-02 15:49:36.37452949 +0000 UTC m=+162.059077403" lastFinishedPulling="2026-06-02 15:49:38.8431938 +0000 UTC m=+164.527741728" observedRunningTime="2026-06-02 15:49:39.48507987 +0000 UTC m=+165.169627801" watchObservedRunningTime="2026-06-02 15:49:39.486181681 +0000 UTC m=+165.170729617"
Jun 02 15:49:39.521387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.521332    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-77c975d575-vh4b6" podStartSLOduration=1.521314717 podStartE2EDuration="1.521314717s" podCreationTimestamp="2026-06-02 15:49:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:39.520802153 +0000 UTC m=+165.205350089" watchObservedRunningTime="2026-06-02 15:49:39.521314717 +0000 UTC m=+165.205862657"
Jun 02 15:49:39.541370 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:39.541319    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-7zqcd" podStartSLOduration=129.060309577 podStartE2EDuration="2m11.541305528s" podCreationTimestamp="2026-06-02 15:47:28 +0000 UTC" firstStartedPulling="2026-06-02 15:49:36.360546952 +0000 UTC m=+162.045094868" lastFinishedPulling="2026-06-02 15:49:38.841542892 +0000 UTC m=+164.526090819" observedRunningTime="2026-06-02 15:49:39.539911907 +0000 UTC m=+165.224459843" watchObservedRunningTime="2026-06-02 15:49:39.541305528 +0000 UTC m=+165.225853463"
Jun 02 15:49:40.966328 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:40.966287    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:40.966742 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:40.966444    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:40.967827 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:40.967800    2568 patch_prober.go:28] interesting pod/console-59666dd688-6g566 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Jun 02 15:49:40.967925 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:40.967848    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59666dd688-6g566" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused"
Jun 02 15:49:47.888856 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:47.888775    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:49:48.820765 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:48.820729    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:48.820962 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:48.820802    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:48.822086 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:48.822046    2568 patch_prober.go:28] interesting pod/console-77c975d575-vh4b6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.14:8443/health\": dial tcp 10.133.0.14:8443: connect: connection refused" start-of-body=
Jun 02 15:49:48.822155 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:48.822106    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-77c975d575-vh4b6" podUID="985837be-0bc6-4b60-8951-629511d04052" containerName="console" probeResult="failure" output="Get \"https://10.133.0.14:8443/health\": dial tcp 10.133.0.14:8443: connect: connection refused"
Jun 02 15:49:49.459975 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:49.459940    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7zqcd"
Jun 02 15:49:50.970919 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:50.970888    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:50.974749 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:50.974724    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:49:56.793101 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:56.793065    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:56.793101 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:56.793106    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:49:58.825020 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:58.824991    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:58.828999 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:58.828977    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:49:58.895538 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:49:58.895504    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:50:16.799087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:16.799042    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:50:16.803027 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:16.803004    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8645bc875b-nbzvh"
Jun 02 15:50:23.169574 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:23.169530    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" podUID="da4f30b5-bc20-46df-a1ea-e322369edf1a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:50:23.916765 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:23.916709    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-59666dd688-6g566" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerName="console" containerID="cri-o://dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559" gracePeriod=15
Jun 02 15:50:24.147356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.147331    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59666dd688-6g566_ae2d9c0b-d147-443d-aaee-56cb893dc2a4/console/0.log"
Jun 02 15:50:24.147469 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.147398    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:50:24.261003 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.260908    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261003 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.260954    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261003 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.260999    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261017    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k6s6\" (UniqueName: \"kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261046    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261087    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261108    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config\") pod \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\" (UID: \"ae2d9c0b-d147-443d-aaee-56cb893dc2a4\") "
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261366    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:24.261566 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261559    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:24.261774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261581    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config" (OuterVolumeSpecName: "console-config") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:24.261774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.261578    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:24.263512 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.263481    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:24.263512 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.263493    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:24.263512 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.263497    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6" (OuterVolumeSpecName: "kube-api-access-7k6s6") pod "ae2d9c0b-d147-443d-aaee-56cb893dc2a4" (UID: "ae2d9c0b-d147-443d-aaee-56cb893dc2a4"). InnerVolumeSpecName "kube-api-access-7k6s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:50:24.362642 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362598    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-oauth-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362642 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362631    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6s6\" (UniqueName: \"kubernetes.io/projected/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-kube-api-access-7k6s6\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362642 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362642    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362642 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362651    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-trusted-ca-bundle\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362903 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362660    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-console-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362903 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362668    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-service-ca\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.362903 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.362676    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae2d9c0b-d147-443d-aaee-56cb893dc2a4-oauth-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:50:24.579867 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579787    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59666dd688-6g566_ae2d9c0b-d147-443d-aaee-56cb893dc2a4/console/0.log"
Jun 02 15:50:24.579867 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579824    2568 generic.go:358] "Generic (PLEG): container finished" podID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerID="dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559" exitCode=2
Jun 02 15:50:24.580087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579857    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59666dd688-6g566" event={"ID":"ae2d9c0b-d147-443d-aaee-56cb893dc2a4","Type":"ContainerDied","Data":"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"}
Jun 02 15:50:24.580087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579899    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59666dd688-6g566"
Jun 02 15:50:24.580087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579904    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59666dd688-6g566" event={"ID":"ae2d9c0b-d147-443d-aaee-56cb893dc2a4","Type":"ContainerDied","Data":"38eed7ddd87f64d7903a98736eebd8d6255d4d6e04d4b87e07729650b73be5cd"}
Jun 02 15:50:24.580087 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.579926    2568 scope.go:117] "RemoveContainer" containerID="dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"
Jun 02 15:50:24.587844 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.587823    2568 scope.go:117] "RemoveContainer" containerID="dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"
Jun 02 15:50:24.588099 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:50:24.588070    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559\": container with ID starting with dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559 not found: ID does not exist" containerID="dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"
Jun 02 15:50:24.588176 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.588105    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559"} err="failed to get container status \"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559\": rpc error: code = NotFound desc = could not find container \"dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559\": container with ID starting with dd98952f6d09c70d13cee2a8bcb30457cdf47f0ff98cc99fd82aa1a75b6ad559 not found: ID does not exist"
Jun 02 15:50:24.603550 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.603525    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:50:24.606964 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.606943    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59666dd688-6g566"]
Jun 02 15:50:24.892489 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:24.892459    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" path="/var/lib/kubelet/pods/ae2d9c0b-d147-443d-aaee-56cb893dc2a4/volumes"
Jun 02 15:50:33.169390 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:33.169336    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" podUID="da4f30b5-bc20-46df-a1ea-e322369edf1a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:50:43.169194 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.169153    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" podUID="da4f30b5-bc20-46df-a1ea-e322369edf1a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 02 15:50:43.169585 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.169229    2568 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7"
Jun 02 15:50:43.169694 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.169676    2568 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6d8d0121e936c72496e7bb0c3124dad6b696ae78546faaef5405f5a82b4251d9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jun 02 15:50:43.169735 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.169712    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" podUID="da4f30b5-bc20-46df-a1ea-e322369edf1a" containerName="service-proxy" containerID="cri-o://6d8d0121e936c72496e7bb0c3124dad6b696ae78546faaef5405f5a82b4251d9" gracePeriod=30
Jun 02 15:50:43.631229 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.631201    2568 generic.go:358] "Generic (PLEG): container finished" podID="da4f30b5-bc20-46df-a1ea-e322369edf1a" containerID="6d8d0121e936c72496e7bb0c3124dad6b696ae78546faaef5405f5a82b4251d9" exitCode=2
Jun 02 15:50:43.631369 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.631252    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerDied","Data":"6d8d0121e936c72496e7bb0c3124dad6b696ae78546faaef5405f5a82b4251d9"}
Jun 02 15:50:43.631369 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:43.631281    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844596bc8d-5m2s7" event={"ID":"da4f30b5-bc20-46df-a1ea-e322369edf1a","Type":"ContainerStarted","Data":"f5c559825aca5a8ebbe25b2043c1ebac76308c58a22e124b5b3a89149700e8b1"}
Jun 02 15:50:59.995187 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:59.995151    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:50:59.995581 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:59.995383    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerName="console"
Jun 02 15:50:59.995581 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:59.995394    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerName="console"
Jun 02 15:50:59.995581 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:59.995438    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae2d9c0b-d147-443d-aaee-56cb893dc2a4" containerName="console"
Jun 02 15:50:59.999126 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:50:59.999093    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.008591 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.008567    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:51:00.139448 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139412    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62l2s\" (UniqueName: \"kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139448 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139446    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139664 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139485    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139664 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139539    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139664 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139596    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139664 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.139664 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.139639    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240352 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240308    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240548 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240362    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240548 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240415    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240548 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240449    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240548 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240475    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240548 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240540    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-62l2s\" (UniqueName: \"kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.240815 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.240570    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.241133 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.241110    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.241298 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.241274    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.241372 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.241357    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.241418 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.241398    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.242912 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.242893    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.242912 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.242902    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.249731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.249687    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62l2s\" (UniqueName: \"kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s\") pod \"console-68c8b87846-pvvfx\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") " pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.307292 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.307271    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:00.433833 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.433803    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:51:00.436757 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:51:00.436722    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a597be_9fee_4ea2_acd6_4aa4d62aa3cb.slice/crio-94ea3bcbbdbc00f02afeaa55d970a77aeeabe6e0b5a978b8bbaf921e48bf7440 WatchSource:0}: Error finding container 94ea3bcbbdbc00f02afeaa55d970a77aeeabe6e0b5a978b8bbaf921e48bf7440: Status 404 returned error can't find the container with id 94ea3bcbbdbc00f02afeaa55d970a77aeeabe6e0b5a978b8bbaf921e48bf7440
Jun 02 15:51:00.674360 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.674328    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8b87846-pvvfx" event={"ID":"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb","Type":"ContainerStarted","Data":"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"}
Jun 02 15:51:00.674360 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.674362    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8b87846-pvvfx" event={"ID":"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb","Type":"ContainerStarted","Data":"94ea3bcbbdbc00f02afeaa55d970a77aeeabe6e0b5a978b8bbaf921e48bf7440"}
Jun 02 15:51:00.694182 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:00.694132    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-68c8b87846-pvvfx" podStartSLOduration=1.694104872 podStartE2EDuration="1.694104872s" podCreationTimestamp="2026-06-02 15:50:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:51:00.693796055 +0000 UTC m=+246.378343991" watchObservedRunningTime="2026-06-02 15:51:00.694104872 +0000 UTC m=+246.378652808"
Jun 02 15:51:06.692195 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:06.692165    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:51:06.694442 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:06.694417    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0186ca7-bafe-4ba4-82bf-0257fbff4133-metrics-certs\") pod \"network-metrics-daemon-xtvfm\" (UID: \"c0186ca7-bafe-4ba4-82bf-0257fbff4133\") " pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:51:06.791815 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:06.791782    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j9fj4\""
Jun 02 15:51:06.799752 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:06.799730    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xtvfm"
Jun 02 15:51:06.922319 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:06.922284    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xtvfm"]
Jun 02 15:51:06.925406 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:51:06.925382    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0186ca7_bafe_4ba4_82bf_0257fbff4133.slice/crio-c619e6b7b2ce074a44b9b2c599793edf8449fc18c724b78976719ead0605c20c WatchSource:0}: Error finding container c619e6b7b2ce074a44b9b2c599793edf8449fc18c724b78976719ead0605c20c: Status 404 returned error can't find the container with id c619e6b7b2ce074a44b9b2c599793edf8449fc18c724b78976719ead0605c20c
Jun 02 15:51:07.693891 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:07.693848    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtvfm" event={"ID":"c0186ca7-bafe-4ba4-82bf-0257fbff4133","Type":"ContainerStarted","Data":"c619e6b7b2ce074a44b9b2c599793edf8449fc18c724b78976719ead0605c20c"}
Jun 02 15:51:08.699957 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:08.699919    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtvfm" event={"ID":"c0186ca7-bafe-4ba4-82bf-0257fbff4133","Type":"ContainerStarted","Data":"962ea14ff10e776aa88800509ad4ad4909ba732a300e02bcdb6f8f1b09552e47"}
Jun 02 15:51:08.699957 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:08.699962    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xtvfm" event={"ID":"c0186ca7-bafe-4ba4-82bf-0257fbff4133","Type":"ContainerStarted","Data":"000374102fb13edc25c64b3d59d4183989a3a3c59846f3b596313520e616f6d8"}
Jun 02 15:51:08.725183 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:08.725137    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xtvfm" podStartSLOduration=253.781492532 podStartE2EDuration="4m14.725123351s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="2026-06-02 15:51:06.927222328 +0000 UTC m=+252.611770243" lastFinishedPulling="2026-06-02 15:51:07.870853146 +0000 UTC m=+253.555401062" observedRunningTime="2026-06-02 15:51:08.724560771 +0000 UTC m=+254.409108705" watchObservedRunningTime="2026-06-02 15:51:08.725123351 +0000 UTC m=+254.409671285"
Jun 02 15:51:10.175647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.175609    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:51:10.211060 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.211015    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:51:10.214193 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.214168    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.227509 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.227485    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:51:10.307974 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.307933    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:10.322322 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322332    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322435 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322393    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwt4\" (UniqueName: \"kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322522 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322478    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322522 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322505    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322608 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322522    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.322608 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.322545    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423623 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423573    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwt4\" (UniqueName: \"kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423766 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423649    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423766 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423675    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423766 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423697    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423910 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423893    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423965 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423940    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.423965 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.423959    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.424576 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.424550    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.424677 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.424595    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.424677 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.424636    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.424814 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.424792    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.426375 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.426323    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.426375 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.426323    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.438759 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.438733    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwt4\" (UniqueName: \"kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4\") pod \"console-5458f54f48-phpng\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") " pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.522937 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.522906    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:10.659369 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.659337    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:51:10.662277 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:51:10.662241    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77fe455_2e54_4c12_8220_120378f5ac35.slice/crio-9225879fdaf1733fec5dfa5758a4fc803c889952cc6f7062d516e827557377f8 WatchSource:0}: Error finding container 9225879fdaf1733fec5dfa5758a4fc803c889952cc6f7062d516e827557377f8: Status 404 returned error can't find the container with id 9225879fdaf1733fec5dfa5758a4fc803c889952cc6f7062d516e827557377f8
Jun 02 15:51:10.706096 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:10.706042    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5458f54f48-phpng" event={"ID":"b77fe455-2e54-4c12-8220-120378f5ac35","Type":"ContainerStarted","Data":"9225879fdaf1733fec5dfa5758a4fc803c889952cc6f7062d516e827557377f8"}
Jun 02 15:51:11.709873 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:11.709835    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5458f54f48-phpng" event={"ID":"b77fe455-2e54-4c12-8220-120378f5ac35","Type":"ContainerStarted","Data":"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"}
Jun 02 15:51:11.737294 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:11.737239    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5458f54f48-phpng" podStartSLOduration=1.7372239839999999 podStartE2EDuration="1.737223984s" podCreationTimestamp="2026-06-02 15:51:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:51:11.73585473 +0000 UTC m=+257.420402665" watchObservedRunningTime="2026-06-02 15:51:11.737223984 +0000 UTC m=+257.421771918"
Jun 02 15:51:20.523156 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:20.523091    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:20.523156 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:20.523163    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:20.527958 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:20.527934    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:20.737925 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:20.737881    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:51:20.828374 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:20.828281    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:51:35.194445 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.194387    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-68c8b87846-pvvfx" podUID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" containerName="console" containerID="cri-o://b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82" gracePeriod=15
Jun 02 15:51:35.435064 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.435029    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68c8b87846-pvvfx_47a597be-9fee-4ea2-acd6-4aa4d62aa3cb/console/0.log"
Jun 02 15:51:35.435161 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.435110    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:35.516219 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516126    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516219 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516168    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516219 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516195    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516477 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516231    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516477 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516265    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516477 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516308    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62l2s\" (UniqueName: \"kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516477 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516345    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert\") pod \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\" (UID: \"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb\") "
Jun 02 15:51:35.516694 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516560    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:35.516782 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516733    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:35.516848 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516802    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:35.516956 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.516930    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config" (OuterVolumeSpecName: "console-config") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:35.518624 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.518596    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s" (OuterVolumeSpecName: "kube-api-access-62l2s") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "kube-api-access-62l2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:51:35.518794 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.518780    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:35.518849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.518808    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" (UID: "47a597be-9fee-4ea2-acd6-4aa4d62aa3cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:35.617023 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.616979    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617023 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617013    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-oauth-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617023 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617023    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-trusted-ca-bundle\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617023 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617032    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62l2s\" (UniqueName: \"kubernetes.io/projected/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-kube-api-access-62l2s\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617304 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617042    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-oauth-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617304 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617082    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-console-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.617304 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.617098    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb-service-ca\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:35.775179 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775098    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68c8b87846-pvvfx_47a597be-9fee-4ea2-acd6-4aa4d62aa3cb/console/0.log"
Jun 02 15:51:35.775179 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775138    2568 generic.go:358] "Generic (PLEG): container finished" podID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" containerID="b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82" exitCode=2
Jun 02 15:51:35.775356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775170    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8b87846-pvvfx" event={"ID":"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb","Type":"ContainerDied","Data":"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"}
Jun 02 15:51:35.775356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775216    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8b87846-pvvfx" event={"ID":"47a597be-9fee-4ea2-acd6-4aa4d62aa3cb","Type":"ContainerDied","Data":"94ea3bcbbdbc00f02afeaa55d970a77aeeabe6e0b5a978b8bbaf921e48bf7440"}
Jun 02 15:51:35.775356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775221    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8b87846-pvvfx"
Jun 02 15:51:35.775356 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.775238    2568 scope.go:117] "RemoveContainer" containerID="b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"
Jun 02 15:51:35.783616 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.783594    2568 scope.go:117] "RemoveContainer" containerID="b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"
Jun 02 15:51:35.783871 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:51:35.783852    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82\": container with ID starting with b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82 not found: ID does not exist" containerID="b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"
Jun 02 15:51:35.783917 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.783878    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82"} err="failed to get container status \"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82\": rpc error: code = NotFound desc = could not find container \"b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82\": container with ID starting with b3d289926fa8f30699f72b2ac7d8ca1227acbdaae329c5b5675fffc6c8e74e82 not found: ID does not exist"
Jun 02 15:51:35.803511 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.803486    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:51:35.807070 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:35.807026    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c8b87846-pvvfx"]
Jun 02 15:51:36.892731 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:36.892699    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" path="/var/lib/kubelet/pods/47a597be-9fee-4ea2-acd6-4aa4d62aa3cb/volumes"
Jun 02 15:51:45.846868 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:45.846803    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-77c975d575-vh4b6" podUID="985837be-0bc6-4b60-8951-629511d04052" containerName="console" containerID="cri-o://636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660" gracePeriod=15
Jun 02 15:51:46.084348 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.084325    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77c975d575-vh4b6_985837be-0bc6-4b60-8951-629511d04052/console/0.log"
Jun 02 15:51:46.084451 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.084384    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:51:46.194372 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194339    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194372 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194376    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194408    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njn4q\" (UniqueName: \"kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194457    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194484    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194511    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194609 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194542    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert\") pod \"985837be-0bc6-4b60-8951-629511d04052\" (UID: \"985837be-0bc6-4b60-8951-629511d04052\") "
Jun 02 15:51:46.194857 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194757    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:46.194905 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.194881    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:46.195026 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.195001    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca" (OuterVolumeSpecName: "service-ca") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:46.195106 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.195007    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config" (OuterVolumeSpecName: "console-config") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:51:46.196700 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.196668    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:46.196803 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.196713    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:51:46.196803 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.196715    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q" (OuterVolumeSpecName: "kube-api-access-njn4q") pod "985837be-0bc6-4b60-8951-629511d04052" (UID: "985837be-0bc6-4b60-8951-629511d04052"). InnerVolumeSpecName "kube-api-access-njn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:51:46.295798 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295748    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.295798 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295789    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-trusted-ca-bundle\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.295798 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295800    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-oauth-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.295798 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295809    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njn4q\" (UniqueName: \"kubernetes.io/projected/985837be-0bc6-4b60-8951-629511d04052-kube-api-access-njn4q\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.295798 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295818    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-console-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.296124 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295826    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/985837be-0bc6-4b60-8951-629511d04052-console-oauth-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.296124 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.295835    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/985837be-0bc6-4b60-8951-629511d04052-service-ca\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:51:46.805403 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805372    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77c975d575-vh4b6_985837be-0bc6-4b60-8951-629511d04052/console/0.log"
Jun 02 15:51:46.805578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805417    2568 generic.go:358] "Generic (PLEG): container finished" podID="985837be-0bc6-4b60-8951-629511d04052" containerID="636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660" exitCode=2
Jun 02 15:51:46.805578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805450    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c975d575-vh4b6" event={"ID":"985837be-0bc6-4b60-8951-629511d04052","Type":"ContainerDied","Data":"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"}
Jun 02 15:51:46.805578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805497    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c975d575-vh4b6"
Jun 02 15:51:46.805578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805512    2568 scope.go:117] "RemoveContainer" containerID="636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"
Jun 02 15:51:46.805759 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.805498    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c975d575-vh4b6" event={"ID":"985837be-0bc6-4b60-8951-629511d04052","Type":"ContainerDied","Data":"f832d0924b1dfbc4b9e430665c52f78056b61cdc2b04121ea287635a80c76248"}
Jun 02 15:51:46.813170 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.813152    2568 scope.go:117] "RemoveContainer" containerID="636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"
Jun 02 15:51:46.813446 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:51:46.813421    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660\": container with ID starting with 636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660 not found: ID does not exist" containerID="636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"
Jun 02 15:51:46.813539 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.813452    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660"} err="failed to get container status \"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660\": rpc error: code = NotFound desc = could not find container \"636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660\": container with ID starting with 636507ded557eda3976df9d7dd57a0edb0f7234d60cb5226585375384c43e660 not found: ID does not exist"
Jun 02 15:51:46.827012 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.826978    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:51:46.830765 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.830745    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77c975d575-vh4b6"]
Jun 02 15:51:46.892544 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:46.892516    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985837be-0bc6-4b60-8951-629511d04052" path="/var/lib/kubelet/pods/985837be-0bc6-4b60-8951-629511d04052/volumes"
Jun 02 15:51:54.779896 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:54.779861    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:51:54.779896 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:54.779901    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:51:54.783974 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:51:54.783952    2568 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:53:05.949831 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.949793    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-6s865"]
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950093    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985837be-0bc6-4b60-8951-629511d04052" containerName="console"
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950106    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="985837be-0bc6-4b60-8951-629511d04052" containerName="console"
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950117    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" containerName="console"
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950124    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" containerName="console"
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950169    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="985837be-0bc6-4b60-8951-629511d04052" containerName="console"
Jun 02 15:53:05.950341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.950179    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="47a597be-9fee-4ea2-acd6-4aa4d62aa3cb" containerName="console"
Jun 02 15:53:05.953139 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.953124    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:05.955469 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.955443    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jun 02 15:53:05.955596 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.955527    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jun 02 15:53:05.955596 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.955558    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jun 02 15:53:05.956395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.956376    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5sct4\""
Jun 02 15:53:05.956503 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.956416    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jun 02 15:53:05.964298 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:05.964275    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6s865"]
Jun 02 15:53:06.044552 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.044516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.044740 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.044625    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8g7c\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-kube-api-access-r8g7c\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.145535 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.145494    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.145697 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.145556    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r8g7c\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-kube-api-access-r8g7c\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.145697 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:53:06.145647    2568 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jun 02 15:53:06.145697 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:53:06.145672    2568 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-6s865: secret "keda-admission-webhooks-certs" not found
Jun 02 15:53:06.145799 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:53:06.145735    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates podName:a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7 nodeName:}" failed. No retries permitted until 2026-06-02 15:53:06.645717432 +0000 UTC m=+372.330265350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates") pod "keda-admission-cf49989db-6s865" (UID: "a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7") : secret "keda-admission-webhooks-certs" not found
Jun 02 15:53:06.156281 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.156251    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8g7c\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-kube-api-access-r8g7c\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.649465 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.649416    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.651774 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.651744    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7-certificates\") pod \"keda-admission-cf49989db-6s865\" (UID: \"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7\") " pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.863113 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.863072    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:06.990489 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.990456    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6s865"]
Jun 02 15:53:06.994843 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:53:06.994815    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a87b2a_b743_40a5_a8c2_7ba21c2ec3b7.slice/crio-ecd12214f7dabfc3bc62ab0e477246022b969cbddc28bef831bdb9fbfff48a6b WatchSource:0}: Error finding container ecd12214f7dabfc3bc62ab0e477246022b969cbddc28bef831bdb9fbfff48a6b: Status 404 returned error can't find the container with id ecd12214f7dabfc3bc62ab0e477246022b969cbddc28bef831bdb9fbfff48a6b
Jun 02 15:53:06.996520 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:06.996505    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:53:07.008937 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:07.008903    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6s865" event={"ID":"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7","Type":"ContainerStarted","Data":"ecd12214f7dabfc3bc62ab0e477246022b969cbddc28bef831bdb9fbfff48a6b"}
Jun 02 15:53:10.018522 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:10.018441    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6s865" event={"ID":"a9a87b2a-b743-40a5-a8c2-7ba21c2ec3b7","Type":"ContainerStarted","Data":"e446a83f1d895bb27549555ee6030a81ce5c18ea3225fddc25248e06fe9bddaf"}
Jun 02 15:53:10.018522 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:10.018485    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:53:10.035892 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:10.035844    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-6s865" podStartSLOduration=2.381268508 podStartE2EDuration="5.035829828s" podCreationTimestamp="2026-06-02 15:53:05 +0000 UTC" firstStartedPulling="2026-06-02 15:53:06.996635177 +0000 UTC m=+372.681183090" lastFinishedPulling="2026-06-02 15:53:09.651196494 +0000 UTC m=+375.335744410" observedRunningTime="2026-06-02 15:53:10.034621536 +0000 UTC m=+375.719169472" watchObservedRunningTime="2026-06-02 15:53:10.035829828 +0000 UTC m=+375.720377763"
Jun 02 15:53:31.024385 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:53:31.024356    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-6s865"
Jun 02 15:54:18.416624 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.416541    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-9r6v9"]
Jun 02 15:54:18.419393 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.419376    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.422295 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.422269    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jun 02 15:54:18.422386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.422359    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jun 02 15:54:18.422729 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.422714    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qkwcd\""
Jun 02 15:54:18.432351 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.432332    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-9r6v9"]
Jun 02 15:54:18.474839 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.474807    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhww\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-kube-api-access-9fhww\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.474968 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.474862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.575747 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.575715    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhww\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-kube-api-access-9fhww\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.575908 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.575771    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.598777 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.598747    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhww\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-kube-api-access-9fhww\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.598970 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.598951    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aee39ae-bfc1-4a76-b8c6-a930e5fa788f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-9r6v9\" (UID: \"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.728093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.727989    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9"
Jun 02 15:54:18.861392 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:18.861365    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-9r6v9"]
Jun 02 15:54:18.863474 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:54:18.863448    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aee39ae_bfc1_4a76_b8c6_a930e5fa788f.slice/crio-18f4dbd0513fb775eb5a229269e792e709ef9df8d69eebdc69c1e4c1c8663e09 WatchSource:0}: Error finding container 18f4dbd0513fb775eb5a229269e792e709ef9df8d69eebdc69c1e4c1c8663e09: Status 404 returned error can't find the container with id 18f4dbd0513fb775eb5a229269e792e709ef9df8d69eebdc69c1e4c1c8663e09
Jun 02 15:54:19.201374 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:19.201338    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9" event={"ID":"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f","Type":"ContainerStarted","Data":"18f4dbd0513fb775eb5a229269e792e709ef9df8d69eebdc69c1e4c1c8663e09"}
Jun 02 15:54:22.211289 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:22.211251    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9" event={"ID":"8aee39ae-bfc1-4a76-b8c6-a930e5fa788f","Type":"ContainerStarted","Data":"9ee69bad5782b2a9405d572753a702c1d21d9db2ecdde6f7da9afef04db76314"}
Jun 02 15:54:22.233076 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:22.233008    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-9r6v9" podStartSLOduration=1.883810097 podStartE2EDuration="4.232990973s" podCreationTimestamp="2026-06-02 15:54:18 +0000 UTC" firstStartedPulling="2026-06-02 15:54:18.865173848 +0000 UTC m=+444.549721761" lastFinishedPulling="2026-06-02 15:54:21.21435471 +0000 UTC m=+446.898902637" observedRunningTime="2026-06-02 15:54:22.231475743 +0000 UTC m=+447.916023680" watchObservedRunningTime="2026-06-02 15:54:22.232990973 +0000 UTC m=+447.917538910"
Jun 02 15:54:31.413518 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.413484    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"]
Jun 02 15:54:31.416536 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.416520    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.418968 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.418941    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:54:31.418968 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.418946    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jun 02 15:54:31.419543 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.419527    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6hdf4\""
Jun 02 15:54:31.427605 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.427587    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"]
Jun 02 15:54:31.473534 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.473506    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f3516f-8dab-48fe-8649-940e3d482501-tmp\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.473650 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.473540    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l55n\" (UniqueName: \"kubernetes.io/projected/24f3516f-8dab-48fe-8649-940e3d482501-kube-api-access-2l55n\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.574462 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.574431    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f3516f-8dab-48fe-8649-940e3d482501-tmp\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.574578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.574468    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2l55n\" (UniqueName: \"kubernetes.io/projected/24f3516f-8dab-48fe-8649-940e3d482501-kube-api-access-2l55n\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.574831 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.574809    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24f3516f-8dab-48fe-8649-940e3d482501-tmp\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.582828 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.582805    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l55n\" (UniqueName: \"kubernetes.io/projected/24f3516f-8dab-48fe-8649-940e3d482501-kube-api-access-2l55n\") pod \"openshift-lws-operator-bfc7f696d-m947k\" (UID: \"24f3516f-8dab-48fe-8649-940e3d482501\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.725015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.724941    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"
Jun 02 15:54:31.845792 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:31.845758    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k"]
Jun 02 15:54:31.849136 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:54:31.849108    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24f3516f_8dab_48fe_8649_940e3d482501.slice/crio-1776b7015b343a8c0e4d12a70a60169a96ab21056f13fc8137611deb4903456a WatchSource:0}: Error finding container 1776b7015b343a8c0e4d12a70a60169a96ab21056f13fc8137611deb4903456a: Status 404 returned error can't find the container with id 1776b7015b343a8c0e4d12a70a60169a96ab21056f13fc8137611deb4903456a
Jun 02 15:54:32.238349 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:32.238315    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k" event={"ID":"24f3516f-8dab-48fe-8649-940e3d482501","Type":"ContainerStarted","Data":"1776b7015b343a8c0e4d12a70a60169a96ab21056f13fc8137611deb4903456a"}
Jun 02 15:54:36.250470 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:36.250377    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k" event={"ID":"24f3516f-8dab-48fe-8649-940e3d482501","Type":"ContainerStarted","Data":"e2c75b96fb6051696e0a7dd39594bd64cb9b4510fddf04338ca2b00ab4ef12a7"}
Jun 02 15:54:36.270631 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:54:36.270576    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-m947k" podStartSLOduration=1.125149787 podStartE2EDuration="5.27055754s" podCreationTimestamp="2026-06-02 15:54:31 +0000 UTC" firstStartedPulling="2026-06-02 15:54:31.850994462 +0000 UTC m=+457.535542375" lastFinishedPulling="2026-06-02 15:54:35.996402211 +0000 UTC m=+461.680950128" observedRunningTime="2026-06-02 15:54:36.268463904 +0000 UTC m=+461.953011839" watchObservedRunningTime="2026-06-02 15:54:36.27055754 +0000 UTC m=+461.955105478"
Jun 02 15:55:05.948862 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:05.948806    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"]
Jun 02 15:55:05.994906 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:05.994875    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"]
Jun 02 15:55:05.995287 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:05.995230    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:05.999514 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:05.999486    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pzsd6\""
Jun 02 15:55:06.002947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.002921    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jun 02 15:55:06.003093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.002949    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jun 02 15:55:06.003093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.003007    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jun 02 15:55:06.131322 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.131276    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-manager-config\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.131532 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.131334    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.131532 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.131377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-metrics-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.131532 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.131401    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pp2\" (UniqueName: \"kubernetes.io/projected/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-kube-api-access-44pp2\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.232207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.232122    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-manager-config\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.232207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.232155    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.232207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.232181    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-metrics-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.232207 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.232196    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44pp2\" (UniqueName: \"kubernetes.io/projected/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-kube-api-access-44pp2\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.232941 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.232918    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-manager-config\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.234564 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.234543    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.234768 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.234748    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-metrics-cert\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.244368 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.244342    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pp2\" (UniqueName: \"kubernetes.io/projected/dfb2eeab-6020-44f9-bd7e-f4c64689aecd-kube-api-access-44pp2\") pod \"lws-controller-manager-74c68556d6-9lvm9\" (UID: \"dfb2eeab-6020-44f9-bd7e-f4c64689aecd\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.307579 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.307551    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:06.436966 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:06.435154    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"]
Jun 02 15:55:06.443178 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:55:06.443141    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb2eeab_6020_44f9_bd7e_f4c64689aecd.slice/crio-c4c08a1d66b68ae0d1eb2f213f32ad96ee61bf94d9645387be301db890a8ed15 WatchSource:0}: Error finding container c4c08a1d66b68ae0d1eb2f213f32ad96ee61bf94d9645387be301db890a8ed15: Status 404 returned error can't find the container with id c4c08a1d66b68ae0d1eb2f213f32ad96ee61bf94d9645387be301db890a8ed15
Jun 02 15:55:07.338108 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:07.338070    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9" event={"ID":"dfb2eeab-6020-44f9-bd7e-f4c64689aecd","Type":"ContainerStarted","Data":"c4c08a1d66b68ae0d1eb2f213f32ad96ee61bf94d9645387be301db890a8ed15"}
Jun 02 15:55:13.354273 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:13.354232    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9" event={"ID":"dfb2eeab-6020-44f9-bd7e-f4c64689aecd","Type":"ContainerStarted","Data":"9bf82fa2a59581ace9a2ef70321e00f398e3b836e86b33e578326b2471844982"}
Jun 02 15:55:13.354675 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:13.354402    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:13.375315 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:13.375271    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9" podStartSLOduration=2.495535534 podStartE2EDuration="8.375258436s" podCreationTimestamp="2026-06-02 15:55:05 +0000 UTC" firstStartedPulling="2026-06-02 15:55:06.445062472 +0000 UTC m=+492.129610399" lastFinishedPulling="2026-06-02 15:55:12.324785385 +0000 UTC m=+498.009333301" observedRunningTime="2026-06-02 15:55:13.374366527 +0000 UTC m=+499.058914463" watchObservedRunningTime="2026-06-02 15:55:13.375258436 +0000 UTC m=+499.059806371"
Jun 02 15:55:24.359184 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:24.359153    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-9lvm9"
Jun 02 15:55:52.094243 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.094162    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"]
Jun 02 15:55:52.097912 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.097891    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:52.102449 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:52.102421    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jun 02 15:55:52.102783 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:52.102755    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-operator-controller-manager-dockercfg-zxbxl\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zxbxl\"" type="*v1.Secret"
Jun 02 15:55:52.103215 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.103185    2568 status_manager.go:895] "Failed to get status for pod" podUID="6bc67b1a-a154-4db0-8694-70287f3aceac" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj" err="pods \"dns-operator-controller-manager-844548ff4c-2xtlj\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object"
Jun 02 15:55:52.103487 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:52.103453    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-operator-controller-env\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" type="*v1.ConfigMap"
Jun 02 15:55:52.103487 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:52.103461    2568 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jun 02 15:55:52.131416 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.131385    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"]
Jun 02 15:55:52.183064 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.183017    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tbq\" (UniqueName: \"kubernetes.io/projected/6bc67b1a-a154-4db0-8694-70287f3aceac-kube-api-access-q8tbq\") pod \"dns-operator-controller-manager-844548ff4c-2xtlj\" (UID: \"6bc67b1a-a154-4db0-8694-70287f3aceac\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:52.284288 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.284255    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tbq\" (UniqueName: \"kubernetes.io/projected/6bc67b1a-a154-4db0-8694-70287f3aceac-kube-api-access-q8tbq\") pod \"dns-operator-controller-manager-844548ff4c-2xtlj\" (UID: \"6bc67b1a-a154-4db0-8694-70287f3aceac\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:52.924078 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:52.924032    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jun 02 15:55:53.324106 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.324003    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zxbxl\""
Jun 02 15:55:53.433020 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.432986    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jun 02 15:55:53.445939 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.445908    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tbq\" (UniqueName: \"kubernetes.io/projected/6bc67b1a-a154-4db0-8694-70287f3aceac-kube-api-access-q8tbq\") pod \"dns-operator-controller-manager-844548ff4c-2xtlj\" (UID: \"6bc67b1a-a154-4db0-8694-70287f3aceac\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:53.587298 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.587201    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jun 02 15:55:53.609362 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.609331    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:53.746366 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:53.746341    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"]
Jun 02 15:55:53.748355 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:55:53.748325    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc67b1a_a154_4db0_8694_70287f3aceac.slice/crio-e89cc69a93df544176cfb7654a4f15685fa399a4cd20459ccdf699ee7d1eced5 WatchSource:0}: Error finding container e89cc69a93df544176cfb7654a4f15685fa399a4cd20459ccdf699ee7d1eced5: Status 404 returned error can't find the container with id e89cc69a93df544176cfb7654a4f15685fa399a4cd20459ccdf699ee7d1eced5
Jun 02 15:55:54.468395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:54.468351    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj" event={"ID":"6bc67b1a-a154-4db0-8694-70287f3aceac","Type":"ContainerStarted","Data":"e89cc69a93df544176cfb7654a4f15685fa399a4cd20459ccdf699ee7d1eced5"}
Jun 02 15:55:57.836763 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.836728    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"]
Jun 02 15:55:57.839795 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.839778    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:57.842847 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.842826    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jun 02 15:55:57.842942 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.842903    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-989ds\""
Jun 02 15:55:57.842942 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.842916    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jun 02 15:55:57.848136 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.848113    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"]
Jun 02 15:55:57.862305 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.862286    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jun 02 15:55:57.930955 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.930930    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xx6\" (UniqueName: \"kubernetes.io/projected/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-kube-api-access-q7xx6\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:57.931066 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.930971    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:57.931130 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:57.931105    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.032540 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.032496    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xx6\" (UniqueName: \"kubernetes.io/projected/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-kube-api-access-q7xx6\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.032716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.032548    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.032716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.032602    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.032835 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:58.032726    2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jun 02 15:55:58.032835 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:55:58.032805    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert podName:8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f nodeName:}" failed. No retries permitted until 2026-06-02 15:55:58.532783336 +0000 UTC m=+544.217331263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-v6vt8" (UID: "8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f") : secret "plugin-serving-cert" not found
Jun 02 15:55:58.033437 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.033416    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.041864 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.041840    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xx6\" (UniqueName: \"kubernetes.io/projected/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-kube-api-access-q7xx6\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.482677 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.482642    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj" event={"ID":"6bc67b1a-a154-4db0-8694-70287f3aceac","Type":"ContainerStarted","Data":"287984b82bcf4f06062d18281fc33a9723695c7a5c3bdc1811c30d465086a643"}
Jun 02 15:55:58.482849 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.482759    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:55:58.509900 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.509852    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj" podStartSLOduration=2.402435934 podStartE2EDuration="6.509839182s" podCreationTimestamp="2026-06-02 15:55:52 +0000 UTC" firstStartedPulling="2026-06-02 15:55:53.750753818 +0000 UTC m=+539.435301731" lastFinishedPulling="2026-06-02 15:55:57.858157067 +0000 UTC m=+543.542704979" observedRunningTime="2026-06-02 15:55:58.507556221 +0000 UTC m=+544.192104156" watchObservedRunningTime="2026-06-02 15:55:58.509839182 +0000 UTC m=+544.194387117"
Jun 02 15:55:58.535983 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.535952    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.538330 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.538303    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-v6vt8\" (UID: \"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.749219 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.749127    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"
Jun 02 15:55:58.879717 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:58.879647    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8"]
Jun 02 15:55:58.882277 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:55:58.882250    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f5aa1ff_a62d_428b_b5b5_8bd5b7e40f7f.slice/crio-d935752318b232bfa65b8003ba23ae8296f6be2f9118e6dad135396c509cbe53 WatchSource:0}: Error finding container d935752318b232bfa65b8003ba23ae8296f6be2f9118e6dad135396c509cbe53: Status 404 returned error can't find the container with id d935752318b232bfa65b8003ba23ae8296f6be2f9118e6dad135396c509cbe53
Jun 02 15:55:59.486600 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:55:59.486562    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8" event={"ID":"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f","Type":"ContainerStarted","Data":"d935752318b232bfa65b8003ba23ae8296f6be2f9118e6dad135396c509cbe53"}
Jun 02 15:56:05.506799 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:05.506761    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8" event={"ID":"8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f","Type":"ContainerStarted","Data":"f30e28bb974cf82520e22358ccd69da04c24c3223600484d16b1ca7c05d52957"}
Jun 02 15:56:05.527432 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:05.527379    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-v6vt8" podStartSLOduration=2.727525639 podStartE2EDuration="8.527365166s" podCreationTimestamp="2026-06-02 15:55:57 +0000 UTC" firstStartedPulling="2026-06-02 15:55:58.88395437 +0000 UTC m=+544.568502290" lastFinishedPulling="2026-06-02 15:56:04.6837939 +0000 UTC m=+550.368341817" observedRunningTime="2026-06-02 15:56:05.525994624 +0000 UTC m=+551.210542572" watchObservedRunningTime="2026-06-02 15:56:05.527365166 +0000 UTC m=+551.211913101"
Jun 02 15:56:06.619028 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.618996    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ccc75765d-xjp79"]
Jun 02 15:56:06.622213 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.622195    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.631922 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.631900    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccc75765d-xjp79"]
Jun 02 15:56:06.707975 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.707946    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-oauth-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708134 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.707982    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708134 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.708000    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kll7\" (UniqueName: \"kubernetes.io/projected/fb2b9c87-cf68-4d56-9c70-815b07e87998-kube-api-access-5kll7\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708134 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.708069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-oauth-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708134 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.708105    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-service-ca\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708273 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.708176    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-trusted-ca-bundle\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.708273 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.708198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808505 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808480    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-trusted-ca-bundle\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808514    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808540    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-oauth-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808569    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808595    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5kll7\" (UniqueName: \"kubernetes.io/projected/fb2b9c87-cf68-4d56-9c70-815b07e87998-kube-api-access-5kll7\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808647 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808629    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-oauth-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.808896 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.808655    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-service-ca\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.809381 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.809353    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.809499 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.809360    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-trusted-ca-bundle\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.809499 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.809394    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-oauth-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.809499 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.809438    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2b9c87-cf68-4d56-9c70-815b07e87998-service-ca\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.811090 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.811074    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-serving-cert\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.811202 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.811187    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb2b9c87-cf68-4d56-9c70-815b07e87998-console-oauth-config\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.818982 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.818964    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kll7\" (UniqueName: \"kubernetes.io/projected/fb2b9c87-cf68-4d56-9c70-815b07e87998-kube-api-access-5kll7\") pod \"console-6ccc75765d-xjp79\" (UID: \"fb2b9c87-cf68-4d56-9c70-815b07e87998\") " pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:06.931171 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:06.931143    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:07.105212 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:07.105188    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccc75765d-xjp79"]
Jun 02 15:56:07.107140 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:56:07.107111    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2b9c87_cf68_4d56_9c70_815b07e87998.slice/crio-cae0f4b7254220aac9522246babd25983f9a72881bf6e6379c437bb35f28aab5 WatchSource:0}: Error finding container cae0f4b7254220aac9522246babd25983f9a72881bf6e6379c437bb35f28aab5: Status 404 returned error can't find the container with id cae0f4b7254220aac9522246babd25983f9a72881bf6e6379c437bb35f28aab5
Jun 02 15:56:07.513597 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:07.513561    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc75765d-xjp79" event={"ID":"fb2b9c87-cf68-4d56-9c70-815b07e87998","Type":"ContainerStarted","Data":"40c397a48cd8137cf98d42bf578d8cb7f50c5632d67770a62dbef8147a35b282"}
Jun 02 15:56:07.513597 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:07.513596    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc75765d-xjp79" event={"ID":"fb2b9c87-cf68-4d56-9c70-815b07e87998","Type":"ContainerStarted","Data":"cae0f4b7254220aac9522246babd25983f9a72881bf6e6379c437bb35f28aab5"}
Jun 02 15:56:09.488629 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:09.488593    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-2xtlj"
Jun 02 15:56:09.516383 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:09.516339    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6ccc75765d-xjp79" podStartSLOduration=3.516326728 podStartE2EDuration="3.516326728s" podCreationTimestamp="2026-06-02 15:56:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:56:07.544810565 +0000 UTC m=+553.229358499" watchObservedRunningTime="2026-06-02 15:56:09.516326728 +0000 UTC m=+555.200874642"
Jun 02 15:56:16.931479 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:16.931442    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:16.931479 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:16.931485    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:16.936147 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:16.936125    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:17.549163 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:17.549131    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ccc75765d-xjp79"
Jun 02 15:56:17.646025 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:17.645995    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:56:40.877132 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.877091    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:40.902036 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.902000    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:40.907681 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.907650    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jun 02 15:56:40.911545 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.911518    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:40.968085 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.968038    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:40.981887 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.981851    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:40.982071 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:40.981910    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxmh\" (UniqueName: \"kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.083365 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.083327    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.083534 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.083374    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxmh\" (UniqueName: \"kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.083992 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.083973    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.098249 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.098217    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxmh\" (UniqueName: \"kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh\") pod \"limitador-limitador-64c8f475fb-k9swx\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.216781 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.216750    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:41.364335 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.364315    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:41.366499 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:56:41.366472    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118acdfc_aa72_47b8_a3e0_2431e7bd13e9.slice/crio-21877476f1743924bf6b8dc3b93982df5b4ffd10dcfcbac000fc11d945ddd09d WatchSource:0}: Error finding container 21877476f1743924bf6b8dc3b93982df5b4ffd10dcfcbac000fc11d945ddd09d: Status 404 returned error can't find the container with id 21877476f1743924bf6b8dc3b93982df5b4ffd10dcfcbac000fc11d945ddd09d
Jun 02 15:56:41.618811 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:41.618717    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" event={"ID":"118acdfc-aa72-47b8-a3e0-2431e7bd13e9","Type":"ContainerStarted","Data":"21877476f1743924bf6b8dc3b93982df5b4ffd10dcfcbac000fc11d945ddd09d"}
Jun 02 15:56:42.664814 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:42.664759    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5458f54f48-phpng" podUID="b77fe455-2e54-4c12-8220-120378f5ac35" containerName="console" containerID="cri-o://744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1" gracePeriod=15
Jun 02 15:56:42.917309 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:42.917251    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5458f54f48-phpng_b77fe455-2e54-4c12-8220-120378f5ac35/console/0.log"
Jun 02 15:56:42.917428 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:42.917328    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:56:43.007710 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007680    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.007710 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007712    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.007921 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007737    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.007921 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007763    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.007921 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007883    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.008078 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007945    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjwt4\" (UniqueName: \"kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.008078 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.007999    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert\") pod \"b77fe455-2e54-4c12-8220-120378f5ac35\" (UID: \"b77fe455-2e54-4c12-8220-120378f5ac35\") "
Jun 02 15:56:43.008188 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008163    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:43.008245 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008179    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca" (OuterVolumeSpecName: "service-ca") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:43.008245 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008167    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config" (OuterVolumeSpecName: "console-config") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:43.008395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008287    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-oauth-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.008395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008291    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:43.008395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008308    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-console-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.008395 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.008320    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-service-ca\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.010264 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.010231    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:56:43.010384 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.010361    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4" (OuterVolumeSpecName: "kube-api-access-jjwt4") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "kube-api-access-jjwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:56:43.010452 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.010430    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b77fe455-2e54-4c12-8220-120378f5ac35" (UID: "b77fe455-2e54-4c12-8220-120378f5ac35"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:56:43.108821 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.108786    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77fe455-2e54-4c12-8220-120378f5ac35-trusted-ca-bundle\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.108821 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.108816    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-oauth-config\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.108821 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.108826    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjwt4\" (UniqueName: \"kubernetes.io/projected/b77fe455-2e54-4c12-8220-120378f5ac35-kube-api-access-jjwt4\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.109032 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.108835    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b77fe455-2e54-4c12-8220-120378f5ac35-console-serving-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:43.627202 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627175    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5458f54f48-phpng_b77fe455-2e54-4c12-8220-120378f5ac35/console/0.log"
Jun 02 15:56:43.627387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627216    2568 generic.go:358] "Generic (PLEG): container finished" podID="b77fe455-2e54-4c12-8220-120378f5ac35" containerID="744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1" exitCode=2
Jun 02 15:56:43.627387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627250    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5458f54f48-phpng" event={"ID":"b77fe455-2e54-4c12-8220-120378f5ac35","Type":"ContainerDied","Data":"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"}
Jun 02 15:56:43.627387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627286    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5458f54f48-phpng"
Jun 02 15:56:43.627387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627295    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5458f54f48-phpng" event={"ID":"b77fe455-2e54-4c12-8220-120378f5ac35","Type":"ContainerDied","Data":"9225879fdaf1733fec5dfa5758a4fc803c889952cc6f7062d516e827557377f8"}
Jun 02 15:56:43.627387 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.627314    2568 scope.go:117] "RemoveContainer" containerID="744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"
Jun 02 15:56:43.654313 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.654275    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:56:43.665381 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.665351    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5458f54f48-phpng"]
Jun 02 15:56:43.706646 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.706503    2568 scope.go:117] "RemoveContainer" containerID="744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"
Jun 02 15:56:43.706800 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:56:43.706778    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1\": container with ID starting with 744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1 not found: ID does not exist" containerID="744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"
Jun 02 15:56:43.706850 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:43.706811    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1"} err="failed to get container status \"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1\": rpc error: code = NotFound desc = could not find container \"744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1\": container with ID starting with 744bf3deb93628fc5503f2292c90cad00e0f65559414f773c19475e11970dbc1 not found: ID does not exist"
Jun 02 15:56:44.631661 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:44.631617    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" event={"ID":"118acdfc-aa72-47b8-a3e0-2431e7bd13e9","Type":"ContainerStarted","Data":"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"}
Jun 02 15:56:44.631862 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:44.631747    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:44.655294 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:44.655234    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" podStartSLOduration=2.263804363 podStartE2EDuration="4.655217169s" podCreationTimestamp="2026-06-02 15:56:40 +0000 UTC" firstStartedPulling="2026-06-02 15:56:41.368349372 +0000 UTC m=+587.052897285" lastFinishedPulling="2026-06-02 15:56:43.759762178 +0000 UTC m=+589.444310091" observedRunningTime="2026-06-02 15:56:44.654383154 +0000 UTC m=+590.338931090" watchObservedRunningTime="2026-06-02 15:56:44.655217169 +0000 UTC m=+590.339765105"
Jun 02 15:56:44.893015 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:44.892939    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77fe455-2e54-4c12-8220-120378f5ac35" path="/var/lib/kubelet/pods/b77fe455-2e54-4c12-8220-120378f5ac35/volumes"
Jun 02 15:56:54.805417 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:54.805381    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:56:54.806428 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:54.806405    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 15:56:55.636764 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:55.636729    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:58.405876 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:58.405847    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:58.406315 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:58.406038    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" podUID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" containerName="limitador" containerID="cri-o://610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481" gracePeriod=30
Jun 02 15:56:58.953550 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:58.953524    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:59.031621 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.031535    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxmh\" (UniqueName: \"kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh\") pod \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") "
Jun 02 15:56:59.031621 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.031591    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file\") pod \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\" (UID: \"118acdfc-aa72-47b8-a3e0-2431e7bd13e9\") "
Jun 02 15:56:59.031928 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.031906    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file" (OuterVolumeSpecName: "config-file") pod "118acdfc-aa72-47b8-a3e0-2431e7bd13e9" (UID: "118acdfc-aa72-47b8-a3e0-2431e7bd13e9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:56:59.033538 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.033518    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh" (OuterVolumeSpecName: "kube-api-access-tjxmh") pod "118acdfc-aa72-47b8-a3e0-2431e7bd13e9" (UID: "118acdfc-aa72-47b8-a3e0-2431e7bd13e9"). InnerVolumeSpecName "kube-api-access-tjxmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:56:59.133069 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.133026    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxmh\" (UniqueName: \"kubernetes.io/projected/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-kube-api-access-tjxmh\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:59.133069 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.133074    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/118acdfc-aa72-47b8-a3e0-2431e7bd13e9-config-file\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:56:59.676389 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.676356    2568 generic.go:358] "Generic (PLEG): container finished" podID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" containerID="610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481" exitCode=0
Jun 02 15:56:59.676836 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.676406    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" event={"ID":"118acdfc-aa72-47b8-a3e0-2431e7bd13e9","Type":"ContainerDied","Data":"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"}
Jun 02 15:56:59.676836 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.676422    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx"
Jun 02 15:56:59.676836 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.676438    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-k9swx" event={"ID":"118acdfc-aa72-47b8-a3e0-2431e7bd13e9","Type":"ContainerDied","Data":"21877476f1743924bf6b8dc3b93982df5b4ffd10dcfcbac000fc11d945ddd09d"}
Jun 02 15:56:59.676836 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.676458    2568 scope.go:117] "RemoveContainer" containerID="610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"
Jun 02 15:56:59.684347 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.684321    2568 scope.go:117] "RemoveContainer" containerID="610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"
Jun 02 15:56:59.684637 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:56:59.684618    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481\": container with ID starting with 610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481 not found: ID does not exist" containerID="610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"
Jun 02 15:56:59.684701 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.684645    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481"} err="failed to get container status \"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481\": rpc error: code = NotFound desc = could not find container \"610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481\": container with ID starting with 610ab9d35e4def2e798af438cbd8e7abd98ff44d464f7075e7a4793fce529481 not found: ID does not exist"
Jun 02 15:56:59.701386 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.701363    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:56:59.710853 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:56:59.710832    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-k9swx"]
Jun 02 15:57:00.893622 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:00.893591    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" path="/var/lib/kubelet/pods/118acdfc-aa72-47b8-a3e0-2431e7bd13e9/volumes"
Jun 02 15:57:37.954743 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.954706    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.954998    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" containerName="limitador"
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.955011    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" containerName="limitador"
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.955019    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77fe455-2e54-4c12-8220-120378f5ac35" containerName="console"
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.955024    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77fe455-2e54-4c12-8220-120378f5ac35" containerName="console"
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.955082    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="b77fe455-2e54-4c12-8220-120378f5ac35" containerName="console"
Jun 02 15:57:37.957157 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.955092    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="118acdfc-aa72-47b8-a3e0-2431e7bd13e9" containerName="limitador"
Jun 02 15:57:37.958080 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.958064    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:37.964831 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.964805    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jun 02 15:57:37.964948 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.964873    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 02 15:57:37.964948 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.964898    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z5jxp\""
Jun 02 15:57:37.964948 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.964898    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 02 15:57:37.972130 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:37.972110    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 15:57:38.051096 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.051041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqd7\" (UniqueName: \"kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.051275 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.051104    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.152459 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.152423    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqd7\" (UniqueName: \"kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.152459 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.152463    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.154893 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.154873    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.164325 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.164299    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqd7\" (UniqueName: \"kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7\") pod \"llmisvc-controller-manager-74846f4db6-jx2gh\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") " pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.267970 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.267882    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:38.391973 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.391948    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 15:57:38.394321 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:57:38.394296    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod326ee9ff_a6ba_4e5d_9370_357e6270c463.slice/crio-42c69bb0b6e1226132cffdcb1587baca74b7eec7002202d574f2bcabc76f3b69 WatchSource:0}: Error finding container 42c69bb0b6e1226132cffdcb1587baca74b7eec7002202d574f2bcabc76f3b69: Status 404 returned error can't find the container with id 42c69bb0b6e1226132cffdcb1587baca74b7eec7002202d574f2bcabc76f3b69
Jun 02 15:57:38.786971 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:38.786940    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" event={"ID":"326ee9ff-a6ba-4e5d-9370-357e6270c463","Type":"ContainerStarted","Data":"42c69bb0b6e1226132cffdcb1587baca74b7eec7002202d574f2bcabc76f3b69"}
Jun 02 15:57:41.801515 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:41.801472    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" event={"ID":"326ee9ff-a6ba-4e5d-9370-357e6270c463","Type":"ContainerStarted","Data":"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"}
Jun 02 15:57:41.801945 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:41.801586    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:57:41.825126 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:57:41.825066    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" podStartSLOduration=2.144250846 podStartE2EDuration="4.82503575s" podCreationTimestamp="2026-06-02 15:57:37 +0000 UTC" firstStartedPulling="2026-06-02 15:57:38.39569154 +0000 UTC m=+644.080239468" lastFinishedPulling="2026-06-02 15:57:41.076476459 +0000 UTC m=+646.761024372" observedRunningTime="2026-06-02 15:57:41.824098701 +0000 UTC m=+647.508646633" watchObservedRunningTime="2026-06-02 15:57:41.82503575 +0000 UTC m=+647.509583685"
Jun 02 15:58:12.806158 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:12.806127    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 15:58:47.595441 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.595364    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-8wvqq"]
Jun 02 15:58:47.598467 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.598447    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:47.600906 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.600889    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jun 02 15:58:47.601011 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.600922    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-89khl\""
Jun 02 15:58:47.607640 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.607610    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-8wvqq"]
Jun 02 15:58:47.789973 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.789929    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:47.790194 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.789993    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcg8\" (UniqueName: \"kubernetes.io/projected/e357b82e-e589-4444-b0e3-46f45ae21be1-kube-api-access-mpcg8\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:47.890923 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.890873    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcg8\" (UniqueName: \"kubernetes.io/projected/e357b82e-e589-4444-b0e3-46f45ae21be1-kube-api-access-mpcg8\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:47.891135 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.890998    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:47.891214 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:58:47.891164    2568 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jun 02 15:58:47.891267 ip-10-0-135-190 kubenswrapper[2568]: E0602 15:58:47.891243    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs podName:e357b82e-e589-4444-b0e3-46f45ae21be1 nodeName:}" failed. No retries permitted until 2026-06-02 15:58:48.391221672 +0000 UTC m=+714.075769589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs") pod "model-serving-api-86f7b4b499-8wvqq" (UID: "e357b82e-e589-4444-b0e3-46f45ae21be1") : secret "model-serving-api-tls" not found
Jun 02 15:58:47.902692 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:47.902203    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcg8\" (UniqueName: \"kubernetes.io/projected/e357b82e-e589-4444-b0e3-46f45ae21be1-kube-api-access-mpcg8\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:48.393578 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.393549    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:48.395986 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.395961    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e357b82e-e589-4444-b0e3-46f45ae21be1-tls-certs\") pod \"model-serving-api-86f7b4b499-8wvqq\" (UID: \"e357b82e-e589-4444-b0e3-46f45ae21be1\") " pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:48.509071 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.509016    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:48.636800 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.636776    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-8wvqq"]
Jun 02 15:58:48.639208 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:58:48.639184    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode357b82e_e589_4444_b0e3_46f45ae21be1.slice/crio-e53bdc30790843f23938e039ddf9ac2f812254d9730099c0ee1b0b42679f10e6 WatchSource:0}: Error finding container e53bdc30790843f23938e039ddf9ac2f812254d9730099c0ee1b0b42679f10e6: Status 404 returned error can't find the container with id e53bdc30790843f23938e039ddf9ac2f812254d9730099c0ee1b0b42679f10e6
Jun 02 15:58:48.640833 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.640816    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:58:48.990752 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:48.990662    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-8wvqq" event={"ID":"e357b82e-e589-4444-b0e3-46f45ae21be1","Type":"ContainerStarted","Data":"e53bdc30790843f23938e039ddf9ac2f812254d9730099c0ee1b0b42679f10e6"}
Jun 02 15:58:50.998783 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:50.998750    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-8wvqq" event={"ID":"e357b82e-e589-4444-b0e3-46f45ae21be1","Type":"ContainerStarted","Data":"4c00a58659688021ca7db6929e399d8f6f5673e86674018b2f663df29aea8c82"}
Jun 02 15:58:50.999195 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:50.998901    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:58:51.038837 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:58:51.038790    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-8wvqq" podStartSLOduration=1.764031828 podStartE2EDuration="4.038777512s" podCreationTimestamp="2026-06-02 15:58:47 +0000 UTC" firstStartedPulling="2026-06-02 15:58:48.640968732 +0000 UTC m=+714.325516648" lastFinishedPulling="2026-06-02 15:58:50.915714418 +0000 UTC m=+716.600262332" observedRunningTime="2026-06-02 15:58:51.037776854 +0000 UTC m=+716.722324789" watchObservedRunningTime="2026-06-02 15:58:51.038777512 +0000 UTC m=+716.723325447"
Jun 02 15:59:02.006351 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:02.006318    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-8wvqq"
Jun 02 15:59:03.550093 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.550042    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7rjr7"]
Jun 02 15:59:03.553341 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.553322    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7rjr7"
Jun 02 15:59:03.555514 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.555494    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 02 15:59:03.555604 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.555545    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pgf2g\""
Jun 02 15:59:03.561936 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.561912    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7rjr7"]
Jun 02 15:59:03.604806 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.604770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgws\" (UniqueName: \"kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws\") pod \"s3-init-7rjr7\" (UID: \"3c9e393d-fadc-4ae8-93ac-254bdef49132\") " pod="kserve/s3-init-7rjr7"
Jun 02 15:59:03.705212 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.705168    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgws\" (UniqueName: \"kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws\") pod \"s3-init-7rjr7\" (UID: \"3c9e393d-fadc-4ae8-93ac-254bdef49132\") " pod="kserve/s3-init-7rjr7"
Jun 02 15:59:03.714102 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.714074    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgws\" (UniqueName: \"kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws\") pod \"s3-init-7rjr7\" (UID: \"3c9e393d-fadc-4ae8-93ac-254bdef49132\") " pod="kserve/s3-init-7rjr7"
Jun 02 15:59:03.862391 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.862295    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7rjr7"
Jun 02 15:59:03.985425 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:03.985394    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7rjr7"]
Jun 02 15:59:03.988982 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:59:03.988960    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9e393d_fadc_4ae8_93ac_254bdef49132.slice/crio-0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62 WatchSource:0}: Error finding container 0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62: Status 404 returned error can't find the container with id 0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62
Jun 02 15:59:04.036233 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:04.036197    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7rjr7" event={"ID":"3c9e393d-fadc-4ae8-93ac-254bdef49132","Type":"ContainerStarted","Data":"0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62"}
Jun 02 15:59:09.054602 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:09.054558    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7rjr7" event={"ID":"3c9e393d-fadc-4ae8-93ac-254bdef49132","Type":"ContainerStarted","Data":"b1ffd24f65875b8d89fef8cead9d7963400b59a9ceace22ab9a4d8118fd78771"}
Jun 02 15:59:09.073406 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:09.073357    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-7rjr7" podStartSLOduration=1.7485804630000001 podStartE2EDuration="6.073339409s" podCreationTimestamp="2026-06-02 15:59:03 +0000 UTC" firstStartedPulling="2026-06-02 15:59:03.990853956 +0000 UTC m=+729.675401877" lastFinishedPulling="2026-06-02 15:59:08.315612911 +0000 UTC m=+734.000160823" observedRunningTime="2026-06-02 15:59:09.072199267 +0000 UTC m=+734.756747203" watchObservedRunningTime="2026-06-02 15:59:09.073339409 +0000 UTC m=+734.757887345"
Jun 02 15:59:12.065307 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:12.065272    2568 generic.go:358] "Generic (PLEG): container finished" podID="3c9e393d-fadc-4ae8-93ac-254bdef49132" containerID="b1ffd24f65875b8d89fef8cead9d7963400b59a9ceace22ab9a4d8118fd78771" exitCode=0
Jun 02 15:59:12.065783 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:12.065354    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7rjr7" event={"ID":"3c9e393d-fadc-4ae8-93ac-254bdef49132","Type":"ContainerDied","Data":"b1ffd24f65875b8d89fef8cead9d7963400b59a9ceace22ab9a4d8118fd78771"}
Jun 02 15:59:13.189294 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:13.189267    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7rjr7"
Jun 02 15:59:13.282912 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:13.282873    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsgws\" (UniqueName: \"kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws\") pod \"3c9e393d-fadc-4ae8-93ac-254bdef49132\" (UID: \"3c9e393d-fadc-4ae8-93ac-254bdef49132\") "
Jun 02 15:59:13.285025 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:13.284995    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws" (OuterVolumeSpecName: "kube-api-access-gsgws") pod "3c9e393d-fadc-4ae8-93ac-254bdef49132" (UID: "3c9e393d-fadc-4ae8-93ac-254bdef49132"). InnerVolumeSpecName "kube-api-access-gsgws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:59:13.383716 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:13.383689    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgws\" (UniqueName: \"kubernetes.io/projected/3c9e393d-fadc-4ae8-93ac-254bdef49132-kube-api-access-gsgws\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 15:59:14.072208 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:14.072166    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7rjr7" event={"ID":"3c9e393d-fadc-4ae8-93ac-254bdef49132","Type":"ContainerDied","Data":"0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62"}
Jun 02 15:59:14.072208 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:14.072195    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7rjr7"
Jun 02 15:59:14.072410 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:14.072199    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa670c5fc93a082dda67a07863a66aedc28000b4638def1dfc7b1e24c445b62"
Jun 02 15:59:42.496958 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.496918    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 15:59:42.497427 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.497409    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9e393d-fadc-4ae8-93ac-254bdef49132" containerName="s3-init"
Jun 02 15:59:42.497474 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.497431    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9e393d-fadc-4ae8-93ac-254bdef49132" containerName="s3-init"
Jun 02 15:59:42.497561 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.497550    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c9e393d-fadc-4ae8-93ac-254bdef49132" containerName="s3-init"
Jun 02 15:59:42.769616 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.769528    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 15:59:42.769769 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.769668    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.772421 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.772392    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 15:59:42.772533 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.772486    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 15:59:42.773253 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.773233    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jun 02 15:59:42.773718 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.773699    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 15:59:42.924751 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.924720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.924947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.924756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.924947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.924788    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.924947 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.924910    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.925186 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.924947    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.925186 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.925006    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfns8\" (UniqueName: \"kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:42.925186 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:42.925077    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025648 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025563    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025648 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025602    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025648 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025624    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025916 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025737    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025916 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025817    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025916 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025841    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.025916 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.025883    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tfns8\" (UniqueName: \"kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.026166 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.026031    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.026218 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.026167    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.026218 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.026195    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.026320 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.026262    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.027885 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.027863    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.028288 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.028267    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.034509 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.034489    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfns8\" (UniqueName: \"kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.080403 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.080370    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 15:59:43.212026 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:43.212000    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 15:59:43.214265 ip-10-0-135-190 kubenswrapper[2568]: W0602 15:59:43.214240    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f14eca4_f5da_45e5_ab42_3f5174ddf787.slice/crio-35fb5ca480e75067040a6b3743691ea701388e430fe4209d9843a50bb2732eb0 WatchSource:0}: Error finding container 35fb5ca480e75067040a6b3743691ea701388e430fe4209d9843a50bb2732eb0: Status 404 returned error can't find the container with id 35fb5ca480e75067040a6b3743691ea701388e430fe4209d9843a50bb2732eb0
Jun 02 15:59:44.163355 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:44.163298    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx" event={"ID":"1f14eca4-f5da-45e5-ab42-3f5174ddf787","Type":"ContainerStarted","Data":"35fb5ca480e75067040a6b3743691ea701388e430fe4209d9843a50bb2732eb0"}
Jun 02 15:59:47.174141 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:47.174088    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx" event={"ID":"1f14eca4-f5da-45e5-ab42-3f5174ddf787","Type":"ContainerStarted","Data":"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"}
Jun 02 15:59:51.324105 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:51.324075    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 15:59:51.324468 ip-10-0-135-190 kubenswrapper[2568]: I0602 15:59:51.324297    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx" podUID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" containerName="storage-initializer" containerID="cri-o://71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04" gracePeriod=30
Jun 02 16:00:21.519864 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.519836    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx_1f14eca4-f5da-45e5-ab42-3f5174ddf787/storage-initializer/0.log"
Jun 02 16:00:21.520196 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.519919    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 16:00:21.619493 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619396    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619493 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619452    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619493 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619493    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619509    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619554    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619590    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfns8\" (UniqueName: \"kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619608    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm\") pod \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\" (UID: \"1f14eca4-f5da-45e5-ab42-3f5174ddf787\") "
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619717    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619752    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home" (OuterVolumeSpecName: "home") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:21.619768 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619725    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:21.620153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619883    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-model-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.620153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619903    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tmp-dir\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.620153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.619916    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-home\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.621722 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.621689    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:00:21.621997 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.621768    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm" (OuterVolumeSpecName: "dshm") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:21.621997 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.621769    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8" (OuterVolumeSpecName: "kube-api-access-tfns8") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "kube-api-access-tfns8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:00:21.656253 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.656225    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f14eca4-f5da-45e5-ab42-3f5174ddf787" (UID: "1f14eca4-f5da-45e5-ab42-3f5174ddf787"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:21.720431 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.720397    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-dshm\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.720431 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.720432    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f14eca4-f5da-45e5-ab42-3f5174ddf787-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.720544 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.720444    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:21.720544 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:21.720456    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfns8\" (UniqueName: \"kubernetes.io/projected/1f14eca4-f5da-45e5-ab42-3f5174ddf787-kube-api-access-tfns8\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:00:22.285945 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.285921    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx_1f14eca4-f5da-45e5-ab42-3f5174ddf787/storage-initializer/0.log"
Jun 02 16:00:22.286126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.285962    2568 generic.go:358] "Generic (PLEG): container finished" podID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" containerID="71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04" exitCode=137
Jun 02 16:00:22.286126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.286044    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"
Jun 02 16:00:22.286126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.286062    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx" event={"ID":"1f14eca4-f5da-45e5-ab42-3f5174ddf787","Type":"ContainerDied","Data":"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"}
Jun 02 16:00:22.286126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.286099    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx" event={"ID":"1f14eca4-f5da-45e5-ab42-3f5174ddf787","Type":"ContainerDied","Data":"35fb5ca480e75067040a6b3743691ea701388e430fe4209d9843a50bb2732eb0"}
Jun 02 16:00:22.286126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.286118    2568 scope.go:117] "RemoveContainer" containerID="71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"
Jun 02 16:00:22.315862 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.315306    2568 scope.go:117] "RemoveContainer" containerID="71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"
Jun 02 16:00:22.315862 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:00:22.315719    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04\": container with ID starting with 71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04 not found: ID does not exist" containerID="71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"
Jun 02 16:00:22.315862 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.315759    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04"} err="failed to get container status \"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04\": rpc error: code = NotFound desc = could not find container \"71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04\": container with ID starting with 71b9612a3df7f07aea15a4ad640c73c3343ea48f55df7029bbf134707d2a2d04 not found: ID does not exist"
Jun 02 16:00:22.327112 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.327069    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 16:00:22.330880 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.330849    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-77fd456ccbrzfjx"]
Jun 02 16:00:22.894251 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:22.894221    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" path="/var/lib/kubelet/pods/1f14eca4-f5da-45e5-ab42-3f5174ddf787/volumes"
Jun 02 16:00:30.943293 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.943256    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:00:30.943748 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.943558    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" containerName="storage-initializer"
Jun 02 16:00:30.943748 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.943573    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" containerName="storage-initializer"
Jun 02 16:00:30.943748 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.943629    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="1f14eca4-f5da-45e5-ab42-3f5174ddf787" containerName="storage-initializer"
Jun 02 16:00:30.948459 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.948441    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.955945 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.955924    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:00:30.956715 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.956700    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:00:30.957285 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.957267    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
Jun 02 16:00:30.957377 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.957310    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:00:30.972026 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.972000    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:00:30.996572 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996537    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996710 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmqw\" (UniqueName: \"kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996710 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996710 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996698    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996818 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996818 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996758    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:30.996892 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:30.996820    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097359 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097321    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097368    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097414    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmqw\" (UniqueName: \"kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097435    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097463    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097497    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097551 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097532    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097843 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097826    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097831    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.097890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097881    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.098010 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.097988    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.099827 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.099807    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.100037 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.100018    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.113298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.113272    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmqw\" (UniqueName: \"kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.258246 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.258157    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:00:31.421257 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:31.421224    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:00:31.425395 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:00:31.425368    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5845a7_ea18_45d4_9358_f66c321230e8.slice/crio-f8e9380ac7508739b8850b4ca46c58976f185adc870565281798a09bd72916bd WatchSource:0}: Error finding container f8e9380ac7508739b8850b4ca46c58976f185adc870565281798a09bd72916bd: Status 404 returned error can't find the container with id f8e9380ac7508739b8850b4ca46c58976f185adc870565281798a09bd72916bd
Jun 02 16:00:32.320888 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:32.320798    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerStarted","Data":"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"}
Jun 02 16:00:32.320888 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:32.320841    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerStarted","Data":"f8e9380ac7508739b8850b4ca46c58976f185adc870565281798a09bd72916bd"}
Jun 02 16:00:35.332519 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:35.332486    2568 generic.go:358] "Generic (PLEG): container finished" podID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerID="2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896" exitCode=0
Jun 02 16:00:35.332844 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:00:35.332565    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerDied","Data":"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"}
Jun 02 16:01:14.455114 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:14.455018    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerStarted","Data":"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"}
Jun 02 16:01:14.475878 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:14.474920    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" podStartSLOduration=5.658031753 podStartE2EDuration="44.474903077s" podCreationTimestamp="2026-06-02 16:00:30 +0000 UTC" firstStartedPulling="2026-06-02 16:00:35.333598938 +0000 UTC m=+821.018146854" lastFinishedPulling="2026-06-02 16:01:14.150470266 +0000 UTC m=+859.835018178" observedRunningTime="2026-06-02 16:01:14.474024635 +0000 UTC m=+860.158572570" watchObservedRunningTime="2026-06-02 16:01:14.474903077 +0000 UTC m=+860.159451013"
Jun 02 16:01:21.259215 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:21.259176    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:21.259680 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:21.259317    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:21.271435 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:21.271411    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:21.488376 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:21.488347    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:31.807841 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:31.807805    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:01:31.808450 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:31.808191    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="main" containerID="cri-o://b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02" gracePeriod=30
Jun 02 16:01:32.053775 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.053750    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:32.095426 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095340    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095445    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095484    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095519    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095553    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095585    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tmqw\" (UniqueName: \"kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095623 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095609    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home\") pod \"dc5845a7-ea18-45d4-9358-f66c321230e8\" (UID: \"dc5845a7-ea18-45d4-9358-f66c321230e8\") "
Jun 02 16:01:32.095955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095738    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:32.095955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095754    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:32.095955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095844    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-model-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.095955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.095861    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-tmp-dir\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.096191 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.096088    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home" (OuterVolumeSpecName: "home") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:32.098461 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.098422    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:32.098758 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.098720    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw" (OuterVolumeSpecName: "kube-api-access-4tmqw") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "kube-api-access-4tmqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:32.098758 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.098726    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm" (OuterVolumeSpecName: "dshm") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:32.152828 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.152786    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc5845a7-ea18-45d4-9358-f66c321230e8" (UID: "dc5845a7-ea18-45d4-9358-f66c321230e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:32.197250 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.197222    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.197250 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.197247    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-dshm\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.197412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.197263    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmqw\" (UniqueName: \"kubernetes.io/projected/dc5845a7-ea18-45d4-9358-f66c321230e8-kube-api-access-4tmqw\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.197412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.197276    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc5845a7-ea18-45d4-9358-f66c321230e8-home\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.197412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.197284    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc5845a7-ea18-45d4-9358-f66c321230e8-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:01:32.513638 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.513607    2568 generic.go:358] "Generic (PLEG): container finished" podID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerID="b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02" exitCode=0
Jun 02 16:01:32.513801 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.513662    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerDied","Data":"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"}
Jun 02 16:01:32.513801 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.513676    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"
Jun 02 16:01:32.513801 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.513691    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd" event={"ID":"dc5845a7-ea18-45d4-9358-f66c321230e8","Type":"ContainerDied","Data":"f8e9380ac7508739b8850b4ca46c58976f185adc870565281798a09bd72916bd"}
Jun 02 16:01:32.513801 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.513707    2568 scope.go:117] "RemoveContainer" containerID="b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"
Jun 02 16:01:32.522512 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.522495    2568 scope.go:117] "RemoveContainer" containerID="2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"
Jun 02 16:01:32.546524 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.546500    2568 scope.go:117] "RemoveContainer" containerID="b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"
Jun 02 16:01:32.546797 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:01:32.546774    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02\": container with ID starting with b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02 not found: ID does not exist" containerID="b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"
Jun 02 16:01:32.546878 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.546811    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02"} err="failed to get container status \"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02\": rpc error: code = NotFound desc = could not find container \"b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02\": container with ID starting with b87968af420fd9f5e9b278997c1fc14c4823bd7385806d5f7b6bc829c4071f02 not found: ID does not exist"
Jun 02 16:01:32.546878 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.546836    2568 scope.go:117] "RemoveContainer" containerID="2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"
Jun 02 16:01:32.547103 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:01:32.547085    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896\": container with ID starting with 2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896 not found: ID does not exist" containerID="2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"
Jun 02 16:01:32.547153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.547110    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896"} err="failed to get container status \"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896\": rpc error: code = NotFound desc = could not find container \"2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896\": container with ID starting with 2f43e31919a70dec35da96e88545a68672a754fe31203b149b67ac2bd6bad896 not found: ID does not exist"
Jun 02 16:01:32.553884 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.553860    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:01:32.559746 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.559724    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-6d857fb89f2mxcd"]
Jun 02 16:01:32.895573 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:32.895540    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" path="/var/lib/kubelet/pods/dc5845a7-ea18-45d4-9358-f66c321230e8/volumes"
Jun 02 16:01:33.278806 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.278724    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:01:33.279041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.279025    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="main"
Jun 02 16:01:33.279041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.279043    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="main"
Jun 02 16:01:33.279232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.279096    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="storage-initializer"
Jun 02 16:01:33.279232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.279103    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="storage-initializer"
Jun 02 16:01:33.279232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.279178    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc5845a7-ea18-45d4-9358-f66c321230e8" containerName="main"
Jun 02 16:01:33.283810 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.283787    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.286380 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.286359    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:01:33.286502 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.286358    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:01:33.286670 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.286642    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:01:33.286790 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.286770    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Jun 02 16:01:33.294003 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.293975    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:01:33.306763 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306736    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.306909 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306783    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.306909 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306810    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.306909 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxpw\" (UniqueName: \"kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.307121 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306917    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.307121 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306957    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.307121 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.306986    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408034 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.407993    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408034 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408035    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408305 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408112    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408305 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408136    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408305 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408266    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408312    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxpw\" (UniqueName: \"kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408367    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408424    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408605 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408499    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408605 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408553    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.408673 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.408631    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.410366 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.410336    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.410520 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.410505    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.418404 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.418381    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxpw\" (UniqueName: \"kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.596132 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.596038    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:01:33.735674 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:33.735642    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:01:33.738954 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:01:33.738913    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6942b744_3ecf_4625_bcc6_30e2cbfb5d58.slice/crio-603b340f63b9f9583ed9a5bcb23248810eb7cc6e174c33946755d46ff059a769 WatchSource:0}: Error finding container 603b340f63b9f9583ed9a5bcb23248810eb7cc6e174c33946755d46ff059a769: Status 404 returned error can't find the container with id 603b340f63b9f9583ed9a5bcb23248810eb7cc6e174c33946755d46ff059a769
Jun 02 16:01:34.522317 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:34.522282    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6" event={"ID":"6942b744-3ecf-4625-bcc6-30e2cbfb5d58","Type":"ContainerStarted","Data":"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"}
Jun 02 16:01:34.522317 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:34.522317    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6" event={"ID":"6942b744-3ecf-4625-bcc6-30e2cbfb5d58","Type":"ContainerStarted","Data":"603b340f63b9f9583ed9a5bcb23248810eb7cc6e174c33946755d46ff059a769"}
Jun 02 16:01:54.825781 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:54.825755    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:01:54.827809 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:54.827792    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:01:59.333789 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.333757    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:01:59.339756 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.339735    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.342227 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.342205    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-89dkc\""
Jun 02 16:01:59.342354 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.342275    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jun 02 16:01:59.349535 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.349512    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:01:59.416220 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.416344 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416249    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.416344 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.416344 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416342    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwdn\" (UniqueName: \"kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.416462 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416375    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.416462 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.416400    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517380 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517352    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517529 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517390    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517529 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517437    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517529 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517474    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517667 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517528    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517667 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517575    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwdn\" (UniqueName: \"kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.517919 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517880    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.518036 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517923    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.518036 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517948    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.518036 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.517993    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.519855 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.519837    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.526074 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.526037    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwdn\" (UniqueName: \"kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.649624 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.649593    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:01:59.780292 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:01:59.780268    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:01:59.782920 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:01:59.782891    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6788fc1c_72cc_4ba2_9013_1566ef547847.slice/crio-275dee01090b8589f067e1504cacce14a8501f364049f46695fad3c855c5d4d5 WatchSource:0}: Error finding container 275dee01090b8589f067e1504cacce14a8501f364049f46695fad3c855c5d4d5: Status 404 returned error can't find the container with id 275dee01090b8589f067e1504cacce14a8501f364049f46695fad3c855c5d4d5
Jun 02 16:02:00.604035 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:00.603999    2568 generic.go:358] "Generic (PLEG): container finished" podID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerID="a9283fa2b256c657f6b5aec85e15b18a59c76a83fdf8d4a991d1229a164b57f6" exitCode=0
Jun 02 16:02:00.604419 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:00.604076    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerDied","Data":"a9283fa2b256c657f6b5aec85e15b18a59c76a83fdf8d4a991d1229a164b57f6"}
Jun 02 16:02:00.604419 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:00.604116    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerStarted","Data":"275dee01090b8589f067e1504cacce14a8501f364049f46695fad3c855c5d4d5"}
Jun 02 16:02:02.611841 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:02.611808    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerStarted","Data":"eba39dc31ccddbb7ac38694dd187cf8229405f99d0cb73a26e8300b69693adee"}
Jun 02 16:02:33.024613 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:33.024581    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:02:34.728861 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.728828    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerStarted","Data":"aaeda27526a50207517f378aa90503496f065b7b531d417f686fb060a2e6bc44"}
Jun 02 16:02:34.729336 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.728937    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main" containerID="cri-o://eba39dc31ccddbb7ac38694dd187cf8229405f99d0cb73a26e8300b69693adee" gracePeriod=30
Jun 02 16:02:34.729336 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.728981    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="tokenizer" containerID="cri-o://aaeda27526a50207517f378aa90503496f065b7b531d417f686fb060a2e6bc44" gracePeriod=30
Jun 02 16:02:34.729336 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.729034    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:02:34.732255 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.732225    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jun 02 16:02:34.749607 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:34.749558    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podStartSLOduration=2.667871261 podStartE2EDuration="35.7495436s" podCreationTimestamp="2026-06-02 16:01:59 +0000 UTC" firstStartedPulling="2026-06-02 16:02:00.605224929 +0000 UTC m=+906.289772842" lastFinishedPulling="2026-06-02 16:02:33.68689726 +0000 UTC m=+939.371445181" observedRunningTime="2026-06-02 16:02:34.748480707 +0000 UTC m=+940.433028643" watchObservedRunningTime="2026-06-02 16:02:34.7495436 +0000 UTC m=+940.434091536"
Jun 02 16:02:35.733508 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:35.733464    2568 generic.go:358] "Generic (PLEG): container finished" podID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerID="eba39dc31ccddbb7ac38694dd187cf8229405f99d0cb73a26e8300b69693adee" exitCode=0
Jun 02 16:02:35.733876 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:35.733534    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerDied","Data":"eba39dc31ccddbb7ac38694dd187cf8229405f99d0cb73a26e8300b69693adee"}
Jun 02 16:02:36.709350 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.709318    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:02:36.711752 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.711736    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.714183 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.714162    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jun 02 16:02:36.726039 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.726017    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:02:36.753481 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753493    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753580    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753614    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcm6\" (UniqueName: \"kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753671    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753709    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.753831 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.753724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854479 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854448    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854479 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854485    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854721 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854640    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854721 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854685    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854828 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854735    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854828 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854770    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-clcm6\" (UniqueName: \"kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854941 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854833    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.854941 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.854931    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.855324 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.855299    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.855324 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.855316    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.855479 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.855365    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.856787 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.856761    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.857149 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.857130    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:36.863829 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:36.863752    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcm6\" (UniqueName: \"kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-njvh6\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:37.022793 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:37.022723    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:02:37.151565 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:37.151535    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:02:37.153843 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:02:37.153815    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6838617e_f20f_4d5b_9920_b321693b064e.slice/crio-15f75730f9409e94150eb2355c98176561488662b7c68430ea338dec55c56c3b WatchSource:0}: Error finding container 15f75730f9409e94150eb2355c98176561488662b7c68430ea338dec55c56c3b: Status 404 returned error can't find the container with id 15f75730f9409e94150eb2355c98176561488662b7c68430ea338dec55c56c3b
Jun 02 16:02:37.742140 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:37.742097    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6" event={"ID":"6838617e-f20f-4d5b-9920-b321693b064e","Type":"ContainerStarted","Data":"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"}
Jun 02 16:02:37.742140 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:37.742139    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6" event={"ID":"6838617e-f20f-4d5b-9920-b321693b064e","Type":"ContainerStarted","Data":"15f75730f9409e94150eb2355c98176561488662b7c68430ea338dec55c56c3b"}
Jun 02 16:02:39.649899 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:39.649866    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:02:44.730280 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:02:44.730243    2568 logging.go:55] [core] [Channel #20 SubChannel #21]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.31:9003", ServerName: "10.133.0.31:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.31:9003: connect: connection refused"
Jun 02 16:02:45.730320 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:45.730277    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.31:9003\" within 1s: context deadline exceeded"
Jun 02 16:02:54.729714 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:02:54.729681    2568 logging.go:55] [core] [Channel #22 SubChannel #23]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.31:9003", ServerName: "10.133.0.31:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.31:9003: connect: connection refused"
Jun 02 16:02:55.729674 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:02:55.729626    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.31:9003\" within 1s: context deadline exceeded"
Jun 02 16:03:04.729799 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:03:04.729763    2568 logging.go:55] [core] [Channel #24 SubChannel #25]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.31:9003", ServerName: "10.133.0.31:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.31:9003: connect: connection refused"
Jun 02 16:03:04.827349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:04.827321    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5_6788fc1c-72cc-4ba2-9013-1566ef547847/tokenizer/0.log"
Jun 02 16:03:04.827942 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:04.827921    2568 generic.go:358] "Generic (PLEG): container finished" podID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerID="aaeda27526a50207517f378aa90503496f065b7b531d417f686fb060a2e6bc44" exitCode=137
Jun 02 16:03:04.828042 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:04.827995    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerDied","Data":"aaeda27526a50207517f378aa90503496f065b7b531d417f686fb060a2e6bc44"}
Jun 02 16:03:05.132519 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.132496    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5_6788fc1c-72cc-4ba2-9013-1566ef547847/tokenizer/0.log"
Jun 02 16:03:05.133188 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.133170    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:03:05.305919 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.305838    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.305919 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.305898    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbwdn\" (UniqueName: \"kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.306174 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.305938    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.306174 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.305960    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.306174 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.305976    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.306174 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.306012    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds\") pod \"6788fc1c-72cc-4ba2-9013-1566ef547847\" (UID: \"6788fc1c-72cc-4ba2-9013-1566ef547847\") "
Jun 02 16:03:05.306379 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.306244    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:03:05.306379 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.306356    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:03:05.306450 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.306408    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:03:05.306642 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.306621    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:03:05.308126 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.308098    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn" (OuterVolumeSpecName: "kube-api-access-fbwdn") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "kube-api-access-fbwdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:03:05.308218 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.308128    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6788fc1c-72cc-4ba2-9013-1566ef547847" (UID: "6788fc1c-72cc-4ba2-9013-1566ef547847"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:03:05.406816 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406791    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbwdn\" (UniqueName: \"kubernetes.io/projected/6788fc1c-72cc-4ba2-9013-1566ef547847-kube-api-access-fbwdn\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.406816 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406815    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6788fc1c-72cc-4ba2-9013-1566ef547847-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.406959 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406826    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.406959 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406835    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.406959 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406846    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.406959 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.406855    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6788fc1c-72cc-4ba2-9013-1566ef547847-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:03:05.729739 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.729691    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.31:9003\" within 1s: context deadline exceeded"
Jun 02 16:03:05.832714 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.832684    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5_6788fc1c-72cc-4ba2-9013-1566ef547847/tokenizer/0.log"
Jun 02 16:03:05.833455 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.833437    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"
Jun 02 16:03:05.833563 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.833448    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5" event={"ID":"6788fc1c-72cc-4ba2-9013-1566ef547847","Type":"ContainerDied","Data":"275dee01090b8589f067e1504cacce14a8501f364049f46695fad3c855c5d4d5"}
Jun 02 16:03:05.833563 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.833493    2568 scope.go:117] "RemoveContainer" containerID="aaeda27526a50207517f378aa90503496f065b7b531d417f686fb060a2e6bc44"
Jun 02 16:03:05.841959 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.841941    2568 scope.go:117] "RemoveContainer" containerID="eba39dc31ccddbb7ac38694dd187cf8229405f99d0cb73a26e8300b69693adee"
Jun 02 16:03:05.849016 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.848994    2568 scope.go:117] "RemoveContainer" containerID="a9283fa2b256c657f6b5aec85e15b18a59c76a83fdf8d4a991d1229a164b57f6"
Jun 02 16:03:05.873435 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.873411    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:03:05.915260 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:05.915232    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-bbc5cbf5zml5"]
Jun 02 16:03:06.893423 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:03:06.893391    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" path="/var/lib/kubelet/pods/6788fc1c-72cc-4ba2-9013-1566ef547847/volumes"
Jun 02 16:06:54.849215 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:06:54.849182    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:06:54.851036 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:06:54.851014    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:11:54.869814 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:11:54.869784    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:11:54.872507 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:11:54.872488    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:16:35.133704 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:35.133666    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:16:35.136272 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:35.134094    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6" podUID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" containerName="storage-initializer" containerID="cri-o://5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302" gracePeriod=30
Jun 02 16:16:39.928960 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.928920    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929237    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="tokenizer"
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929250    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="tokenizer"
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929261    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="storage-initializer"
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929267    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="storage-initializer"
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929275    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main"
Jun 02 16:16:39.929357 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929312    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main"
Jun 02 16:16:39.929637 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929380    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="main"
Jun 02 16:16:39.929637 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.929388    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6788fc1c-72cc-4ba2-9013-1566ef547847" containerName="tokenizer"
Jun 02 16:16:39.932591 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.932568    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:39.934862 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.934841    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jun 02 16:16:39.935442 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.935427    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-mjv9x\""
Jun 02 16:16:39.946618 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:39.946594    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:16:40.035107 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.035107 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035108    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9ps\" (UniqueName: \"kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.035324 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035136    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.035324 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035226    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.035324 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035312    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.035444 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.035429    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.136627 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136590    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.136627 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136632    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9ps\" (UniqueName: \"kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.136890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136663    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.136890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136797    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.136890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136853    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.137082 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.136970    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.137264 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.137239    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.137389 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.137295    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.137389 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.137357    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.137389 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.137378    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.139371 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.139353    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.145220 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.145200    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9ps\" (UniqueName: \"kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps\") pod \"custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.241508 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.241427    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:40.375403 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.375380    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:16:40.377447 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:16:40.377414    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4365829d_3df6_4ffa_9423_4feba48b830e.slice/crio-38ecc43fde5edcde29def05669f20bc253626612cc35110582334adcaa9f0b51 WatchSource:0}: Error finding container 38ecc43fde5edcde29def05669f20bc253626612cc35110582334adcaa9f0b51: Status 404 returned error can't find the container with id 38ecc43fde5edcde29def05669f20bc253626612cc35110582334adcaa9f0b51
Jun 02 16:16:40.379232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:40.379212    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:16:41.377878 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:41.377842    2568 generic.go:358] "Generic (PLEG): container finished" podID="4365829d-3df6-4ffa-9423-4feba48b830e" containerID="4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f" exitCode=0
Jun 02 16:16:41.378367 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:41.377934    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerDied","Data":"4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f"}
Jun 02 16:16:41.378367 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:41.377972    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerStarted","Data":"38ecc43fde5edcde29def05669f20bc253626612cc35110582334adcaa9f0b51"}
Jun 02 16:16:42.383069 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:42.382971    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerStarted","Data":"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"}
Jun 02 16:16:42.383069 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:42.383002    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerStarted","Data":"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"}
Jun 02 16:16:42.383495 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:42.383201    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:42.411060 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:42.410999    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" podStartSLOduration=3.410980263 podStartE2EDuration="3.410980263s" podCreationTimestamp="2026-06-02 16:16:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:16:42.406149126 +0000 UTC m=+1788.090697061" watchObservedRunningTime="2026-06-02 16:16:42.410980263 +0000 UTC m=+1788.095528193"
Jun 02 16:16:50.242343 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:50.242307    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:50.242819 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:50.242467    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:50.245101 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:50.245075    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:50.411853 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:50.411822    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:16:54.891171 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:54.891144    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:16:54.895499 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:16:54.895477    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:17:05.331589 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.331563    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6_6942b744-3ecf-4625-bcc6-30e2cbfb5d58/storage-initializer/0.log"
Jun 02 16:17:05.331891 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.331631    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:17:05.454470 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454377    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfxpw\" (UniqueName: \"kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454619 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454499    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454619 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454554    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454730 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454617    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454730 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454646    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454730 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454684    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454730 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454707    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home\") pod \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\" (UID: \"6942b744-3ecf-4625-bcc6-30e2cbfb5d58\") "
Jun 02 16:17:05.454933 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454786    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:05.454933 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.454827    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache" (OuterVolumeSpecName: "model-cache") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:05.455042 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.455006    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-model-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.455042 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.455026    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tmp-dir\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.455042 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.455038    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home" (OuterVolumeSpecName: "home") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:05.456620 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.456598    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm" (OuterVolumeSpecName: "dshm") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:05.456751 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.456674    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:17:05.456751 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.456697    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw" (OuterVolumeSpecName: "kube-api-access-rfxpw") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "kube-api-access-rfxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:17:05.462914 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.462895    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6_6942b744-3ecf-4625-bcc6-30e2cbfb5d58/storage-initializer/0.log"
Jun 02 16:17:05.463007 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.462935    2568 generic.go:358] "Generic (PLEG): container finished" podID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" containerID="5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302" exitCode=137
Jun 02 16:17:05.463007 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.462985    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6" event={"ID":"6942b744-3ecf-4625-bcc6-30e2cbfb5d58","Type":"ContainerDied","Data":"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"}
Jun 02 16:17:05.463128 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.463014    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"
Jun 02 16:17:05.463128 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.463031    2568 scope.go:117] "RemoveContainer" containerID="5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"
Jun 02 16:17:05.463233 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.463018    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6" event={"ID":"6942b744-3ecf-4625-bcc6-30e2cbfb5d58","Type":"ContainerDied","Data":"603b340f63b9f9583ed9a5bcb23248810eb7cc6e174c33946755d46ff059a769"}
Jun 02 16:17:05.474231 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.474206    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6942b744-3ecf-4625-bcc6-30e2cbfb5d58" (UID: "6942b744-3ecf-4625-bcc6-30e2cbfb5d58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:05.492955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.492931    2568 scope.go:117] "RemoveContainer" containerID="5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"
Jun 02 16:17:05.493253 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:17:05.493233    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302\": container with ID starting with 5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302 not found: ID does not exist" containerID="5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"
Jun 02 16:17:05.493331 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.493262    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302"} err="failed to get container status \"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302\": rpc error: code = NotFound desc = could not find container \"5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302\": container with ID starting with 5dab8c169eaabbc8aaa8f46e612973b182ac4c664932aa44f2c78eab86025302 not found: ID does not exist"
Jun 02 16:17:05.555732 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.555693    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxpw\" (UniqueName: \"kubernetes.io/projected/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kube-api-access-rfxpw\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.555732 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.555727    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.555732 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.555738    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-dshm\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.555955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.555749    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.555955 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.555758    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6942b744-3ecf-4625-bcc6-30e2cbfb5d58-home\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:17:05.821156 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.821119    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:17:05.825072 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:05.825017    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-cf5d46fcf-72sp6"]
Jun 02 16:17:06.893291 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:06.893258    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" path="/var/lib/kubelet/pods/6942b744-3ecf-4625-bcc6-30e2cbfb5d58/volumes"
Jun 02 16:17:12.418695 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:12.418664    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:17:36.335635 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:36.335600    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:17:36.336073 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:17:36.335897    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6" podUID="6838617e-f20f-4d5b-9920-b321693b064e" containerName="storage-initializer" containerID="cri-o://d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584" gracePeriod=30
Jun 02 16:18:06.528927 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.528896    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-6879b9c49c-njvh6_6838617e-f20f-4d5b-9920-b321693b064e/storage-initializer/0.log"
Jun 02 16:18:06.529381 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.528998    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:18:06.568864 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.568835    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.568885    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.568939    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.568966    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.568984    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569005    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clcm6\" (UniqueName: \"kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569041 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569031    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs\") pod \"6838617e-f20f-4d5b-9920-b321693b064e\" (UID: \"6838617e-f20f-4d5b-9920-b321693b064e\") "
Jun 02 16:18:06.569349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569188    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home" (OuterVolumeSpecName: "home") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:06.569349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569209    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:06.569349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569263    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:06.569349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569300    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-home\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.569349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.569321    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-tmp-dir\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.571119 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.571089    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm" (OuterVolumeSpecName: "dshm") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:06.571221 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.571193    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:18:06.571278 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.571223    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6" (OuterVolumeSpecName: "kube-api-access-clcm6") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "kube-api-access-clcm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:18:06.605163 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.605097    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6838617e-f20f-4d5b-9920-b321693b064e" (UID: "6838617e-f20f-4d5b-9920-b321693b064e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:18:06.656632 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656604    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-6879b9c49c-njvh6_6838617e-f20f-4d5b-9920-b321693b064e/storage-initializer/0.log"
Jun 02 16:18:06.656759 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656642    2568 generic.go:358] "Generic (PLEG): container finished" podID="6838617e-f20f-4d5b-9920-b321693b064e" containerID="d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584" exitCode=137
Jun 02 16:18:06.656759 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656717    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"
Jun 02 16:18:06.656759 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656735    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6" event={"ID":"6838617e-f20f-4d5b-9920-b321693b064e","Type":"ContainerDied","Data":"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"}
Jun 02 16:18:06.656860 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656775    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6" event={"ID":"6838617e-f20f-4d5b-9920-b321693b064e","Type":"ContainerDied","Data":"15f75730f9409e94150eb2355c98176561488662b7c68430ea338dec55c56c3b"}
Jun 02 16:18:06.656860 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.656792    2568 scope.go:117] "RemoveContainer" containerID="d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"
Jun 02 16:18:06.670422 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.670399    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-model-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.670514 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.670430    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.670514 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.670447    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6838617e-f20f-4d5b-9920-b321693b064e-dshm\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.670514 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.670463    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcm6\" (UniqueName: \"kubernetes.io/projected/6838617e-f20f-4d5b-9920-b321693b064e-kube-api-access-clcm6\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.670514 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.670480    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6838617e-f20f-4d5b-9920-b321693b064e-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.679785 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.679763    2568 scope.go:117] "RemoveContainer" containerID="d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"
Jun 02 16:18:06.680044 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:18:06.680022    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584\": container with ID starting with d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584 not found: ID does not exist" containerID="d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"
Jun 02 16:18:06.680115 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.680069    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584"} err="failed to get container status \"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584\": rpc error: code = NotFound desc = could not find container \"d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584\": container with ID starting with d1d22140a4f56fb66392a366ace50b7297e2e30a1e78456f67b8190d026b6584 not found: ID does not exist"
Jun 02 16:18:06.704724 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.704693    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:18:06.709488 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.709454    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-njvh6"]
Jun 02 16:18:06.893367 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:18:06.893334    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6838617e-f20f-4d5b-9920-b321693b064e" path="/var/lib/kubelet/pods/6838617e-f20f-4d5b-9920-b321693b064e/volumes"
Jun 02 16:19:49.479559 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:49.479525    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:19:49.480095 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:49.479959    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="main" containerID="cri-o://efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec" gracePeriod=30
Jun 02 16:19:49.480095 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:49.480011    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="tokenizer" containerID="cri-o://cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d" gracePeriod=30
Jun 02 16:19:49.995888 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:49.995854    2568 generic.go:358] "Generic (PLEG): container finished" podID="4365829d-3df6-4ffa-9423-4feba48b830e" containerID="efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec" exitCode=0
Jun 02 16:19:49.996138 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:49.995930    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerDied","Data":"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"}
Jun 02 16:19:50.411401 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.411357    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="tokenizer" probeResult="failure" output="Get \"http://10.133.0.33:8082/healthz\": dial tcp 10.133.0.33:8082: connect: connection refused"
Jun 02 16:19:50.641459 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.641436    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:19:50.729466 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729398    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729466 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729454    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729670 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729508    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz9ps\" (UniqueName: \"kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729670 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729541    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729670 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729592    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729670 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729625    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs\") pod \"4365829d-3df6-4ffa-9423-4feba48b830e\" (UID: \"4365829d-3df6-4ffa-9423-4feba48b830e\") "
Jun 02 16:19:50.729875 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729758    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:19:50.729875 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729832    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:19:50.729990 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729907    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:50.729990 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.729945    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:19:50.730194 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.730177    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:19:50.731625 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.731601    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps" (OuterVolumeSpecName: "kube-api-access-kz9ps") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "kube-api-access-kz9ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:19:50.731719 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.731695    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4365829d-3df6-4ffa-9423-4feba48b830e" (UID: "4365829d-3df6-4ffa-9423-4feba48b830e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:19:50.830561 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.830526    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9ps\" (UniqueName: \"kubernetes.io/projected/4365829d-3df6-4ffa-9423-4feba48b830e-kube-api-access-kz9ps\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:50.830561 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.830556    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:50.830561 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.830566    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:50.830774 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.830576    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4365829d-3df6-4ffa-9423-4feba48b830e-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:50.830774 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:50.830584    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4365829d-3df6-4ffa-9423-4feba48b830e-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:19:51.000731 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.000633    2568 generic.go:358] "Generic (PLEG): container finished" podID="4365829d-3df6-4ffa-9423-4feba48b830e" containerID="cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d" exitCode=0
Jun 02 16:19:51.000731 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.000689    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerDied","Data":"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"}
Jun 02 16:19:51.000731 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.000717    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"
Jun 02 16:19:51.000989 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.000733    2568 scope.go:117] "RemoveContainer" containerID="cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"
Jun 02 16:19:51.000989 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.000719    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw" event={"ID":"4365829d-3df6-4ffa-9423-4feba48b830e","Type":"ContainerDied","Data":"38ecc43fde5edcde29def05669f20bc253626612cc35110582334adcaa9f0b51"}
Jun 02 16:19:51.010807 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.010777    2568 scope.go:117] "RemoveContainer" containerID="efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"
Jun 02 16:19:51.020036 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.020010    2568 scope.go:117] "RemoveContainer" containerID="4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f"
Jun 02 16:19:51.021681 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.021604    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:19:51.024066 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.024027    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-f48c74658gmvw"]
Jun 02 16:19:51.027747 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.027727    2568 scope.go:117] "RemoveContainer" containerID="cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"
Jun 02 16:19:51.027981 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:19:51.027963    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d\": container with ID starting with cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d not found: ID does not exist" containerID="cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"
Jun 02 16:19:51.028031 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.027990    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d"} err="failed to get container status \"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d\": rpc error: code = NotFound desc = could not find container \"cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d\": container with ID starting with cb63fbfe58a36d2489d82e5ce8c14f610aff6e607d4f950b73623b7dc917928d not found: ID does not exist"
Jun 02 16:19:51.028031 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.028009    2568 scope.go:117] "RemoveContainer" containerID="efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"
Jun 02 16:19:51.028276 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:19:51.028257    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec\": container with ID starting with efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec not found: ID does not exist" containerID="efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"
Jun 02 16:19:51.028320 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.028282    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec"} err="failed to get container status \"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec\": rpc error: code = NotFound desc = could not find container \"efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec\": container with ID starting with efad6cac44e2fc7fc46388fdd7f2e8aeb1c32221b4b2153278b631292634d4ec not found: ID does not exist"
Jun 02 16:19:51.028320 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.028299    2568 scope.go:117] "RemoveContainer" containerID="4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f"
Jun 02 16:19:51.028522 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:19:51.028507    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f\": container with ID starting with 4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f not found: ID does not exist" containerID="4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f"
Jun 02 16:19:51.028564 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:51.028525    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f"} err="failed to get container status \"4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f\": rpc error: code = NotFound desc = could not find container \"4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f\": container with ID starting with 4395b4aa89662ead65d4adac679fb9c01619fa4a54fbb1630e5dcacebca8af0f not found: ID does not exist"
Jun 02 16:19:52.893237 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:19:52.893210    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" path="/var/lib/kubelet/pods/4365829d-3df6-4ffa-9423-4feba48b830e/volumes"
Jun 02 16:20:02.587864 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.587828    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:20:02.588276 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588261    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="storage-initializer"
Jun 02 16:20:02.588319 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588279    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="storage-initializer"
Jun 02 16:20:02.588319 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588299    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6838617e-f20f-4d5b-9920-b321693b064e" containerName="storage-initializer"
Jun 02 16:20:02.588319 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588307    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6838617e-f20f-4d5b-9920-b321693b064e" containerName="storage-initializer"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588321    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="tokenizer"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588330    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="tokenizer"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588346    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" containerName="storage-initializer"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588355    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" containerName="storage-initializer"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588367    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="main"
Jun 02 16:20:02.588412 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588375    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="main"
Jun 02 16:20:02.588584 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588444    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6942b744-3ecf-4625-bcc6-30e2cbfb5d58" containerName="storage-initializer"
Jun 02 16:20:02.588584 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588456    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="main"
Jun 02 16:20:02.588584 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588469    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6838617e-f20f-4d5b-9920-b321693b064e" containerName="storage-initializer"
Jun 02 16:20:02.588584 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.588481    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4365829d-3df6-4ffa-9423-4feba48b830e" containerName="tokenizer"
Jun 02 16:20:02.593180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.593154    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.596001 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.595979    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:20:02.596599 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.596579    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jun 02 16:20:02.596738 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.596588    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-pvzzq\""
Jun 02 16:20:02.596738 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.596711    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:20:02.597108 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.597093    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:20:02.609016 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.608990    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:20:02.617878 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.617853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.617974 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.617896    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.617974 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.617931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.618045 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.617987    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.618045 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.618031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.618168 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.618074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pzj\" (UniqueName: \"kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719271 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719229    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719271 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719274    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719296    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719323    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719353    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719377    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pzj\" (UniqueName: \"kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719716 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719688    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719785 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719740    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719847 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719786    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.719847 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.719792    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.721713 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.721696    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.728309 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.728282    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pzj\" (UniqueName: \"kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj\") pod \"router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:02.903249 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:02.903213    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:03.039582 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:03.039543    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:20:03.043060 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:20:03.043020    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610ae4f2_0831_472e_96c1_ad0d1fa8bf6a.slice/crio-0ed9ef7dcecc996063ee7810026f850598004cb72014af392ea32d2a74f8fac9 WatchSource:0}: Error finding container 0ed9ef7dcecc996063ee7810026f850598004cb72014af392ea32d2a74f8fac9: Status 404 returned error can't find the container with id 0ed9ef7dcecc996063ee7810026f850598004cb72014af392ea32d2a74f8fac9
Jun 02 16:20:04.045164 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:04.045082    2568 generic.go:358] "Generic (PLEG): container finished" podID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerID="db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45" exitCode=0
Jun 02 16:20:04.045164 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:04.045146    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerDied","Data":"db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45"}
Jun 02 16:20:04.045564 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:04.045168    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerStarted","Data":"0ed9ef7dcecc996063ee7810026f850598004cb72014af392ea32d2a74f8fac9"}
Jun 02 16:20:05.050454 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:05.050420    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerStarted","Data":"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"}
Jun 02 16:20:05.050454 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:05.050456    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerStarted","Data":"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"}
Jun 02 16:20:05.050849 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:05.050597    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:05.071101 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:05.071013    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" podStartSLOduration=3.070998262 podStartE2EDuration="3.070998262s" podCreationTimestamp="2026-06-02 16:20:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:20:05.069518577 +0000 UTC m=+1990.754066509" watchObservedRunningTime="2026-06-02 16:20:05.070998262 +0000 UTC m=+1990.755546197"
Jun 02 16:20:12.904194 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:12.904160    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:12.904618 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:12.904202    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:12.907022 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:12.906997    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:13.077763 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:13.077733    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:20:34.081225 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:20:34.081196    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:21:54.911266 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:21:54.911238    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:21:54.916984 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:21:54.916961    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:26:54.930344 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:26:54.930311    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:26:54.937385 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:26:54.937364    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:31:54.951589 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:31:54.951470    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:31:54.959103 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:31:54.959082    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:34:53.110131 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:53.110032    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:34:53.110646 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:53.110451    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="main" containerID="cri-o://4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd" gracePeriod=30
Jun 02 16:34:53.110646 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:53.110519    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="tokenizer" containerID="cri-o://2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b" gracePeriod=30
Jun 02 16:34:53.877887 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:53.877848    2568 generic.go:358] "Generic (PLEG): container finished" podID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerID="4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd" exitCode=0
Jun 02 16:34:53.878080 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:53.877893    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerDied","Data":"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"}
Jun 02 16:34:54.080252 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:34:54.080224    2568 logging.go:55] [core] [Channel #653 SubChannel #654]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.34:9003", ServerName: "10.133.0.34:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.34:9003: connect: connection refused"
Jun 02 16:34:54.263447 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.263417    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:34:54.303278 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303244    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5pzj\" (UniqueName: \"kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303452 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303296    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303452 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303315    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303452 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303346    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303452 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303387    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303452 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303428    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location\") pod \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\" (UID: \"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a\") "
Jun 02 16:34:54.303834 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303569    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:34:54.303834 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303664    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.303834 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303674    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:34:54.303834 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.303736    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:34:54.304180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.304158    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:34:54.305380 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.305357    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj" (OuterVolumeSpecName: "kube-api-access-q5pzj") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "kube-api-access-q5pzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:34:54.305469 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.305444    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" (UID: "610ae4f2-0831-472e-96c1-ad0d1fa8bf6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:34:54.404129 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.404097    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pzj\" (UniqueName: \"kubernetes.io/projected/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kube-api-access-q5pzj\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.404129 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.404124    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.404129 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.404136    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.404345 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.404146    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.404345 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.404154    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:34:54.883448 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.883413    2568 generic.go:358] "Generic (PLEG): container finished" podID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerID="2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b" exitCode=0
Jun 02 16:34:54.883613 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.883455    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerDied","Data":"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"}
Jun 02 16:34:54.883613 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.883479    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" event={"ID":"610ae4f2-0831-472e-96c1-ad0d1fa8bf6a","Type":"ContainerDied","Data":"0ed9ef7dcecc996063ee7810026f850598004cb72014af392ea32d2a74f8fac9"}
Jun 02 16:34:54.883613 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.883487    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"
Jun 02 16:34:54.883613 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.883494    2568 scope.go:117] "RemoveContainer" containerID="2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"
Jun 02 16:34:54.892499 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.892471    2568 scope.go:117] "RemoveContainer" containerID="4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"
Jun 02 16:34:54.899392 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.899373    2568 scope.go:117] "RemoveContainer" containerID="db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45"
Jun 02 16:34:54.905570 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.905537    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:34:54.907205 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907186    2568 scope.go:117] "RemoveContainer" containerID="2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"
Jun 02 16:34:54.907419 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907397    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz"]
Jun 02 16:34:54.907520 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:34:54.907460    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b\": container with ID starting with 2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b not found: ID does not exist" containerID="2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"
Jun 02 16:34:54.907520 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907485    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b"} err="failed to get container status \"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b\": rpc error: code = NotFound desc = could not find container \"2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b\": container with ID starting with 2a7396e37367b5024eb61e11763aa864b5d323c5fcb42458e85aa1d43c4ea47b not found: ID does not exist"
Jun 02 16:34:54.907520 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907503    2568 scope.go:117] "RemoveContainer" containerID="4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"
Jun 02 16:34:54.907790 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:34:54.907768    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd\": container with ID starting with 4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd not found: ID does not exist" containerID="4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"
Jun 02 16:34:54.907886 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907795    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd"} err="failed to get container status \"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd\": rpc error: code = NotFound desc = could not find container \"4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd\": container with ID starting with 4fbd266862e21aed98d154c328e14b0cabbba3b1c00bdace32083a5de571d9fd not found: ID does not exist"
Jun 02 16:34:54.907886 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.907810    2568 scope.go:117] "RemoveContainer" containerID="db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45"
Jun 02 16:34:54.908096 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:34:54.908078    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45\": container with ID starting with db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45 not found: ID does not exist" containerID="db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45"
Jun 02 16:34:54.908153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:54.908100    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45"} err="failed to get container status \"db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45\": rpc error: code = NotFound desc = could not find container \"db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45\": container with ID starting with db2b899aca0ba89779b4ddaaef0620e2c6ab2629bd1646d3ff399ff09522db45 not found: ID does not exist"
Jun 02 16:34:55.080446 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:55.080399    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-867cc8b888-t4vjz" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.34:9003\" within 1s: context deadline exceeded"
Jun 02 16:34:56.892145 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:34:56.892115    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" path="/var/lib/kubelet/pods/610ae4f2-0831-472e-96c1-ad0d1fa8bf6a/volumes"
Jun 02 16:35:00.060923 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.060886    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061376    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="main"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061396    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="main"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061435    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="storage-initializer"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061446    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="storage-initializer"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061498    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="tokenizer"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061508    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="tokenizer"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061605    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="main"
Jun 02 16:35:00.061859 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.061620    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="610ae4f2-0831-472e-96c1-ad0d1fa8bf6a" containerName="tokenizer"
Jun 02 16:35:00.064653 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.064638    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.067216 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.067038    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-tn4vj\""
Jun 02 16:35:00.067216 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.067165    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:35:00.067216 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.067177    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:35:00.067753 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.067731    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:35:00.067846 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.067742    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jun 02 16:35:00.074136 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.074111    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:35:00.142497 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142461    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.142657 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142522    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.142657 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142570    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.142657 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142633    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rdg\" (UniqueName: \"kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.142763 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.142763 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.142694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243447 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243406    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243447 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243448    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243704 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243477    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243704 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243539    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243704 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243568    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243704 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243608    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rdg\" (UniqueName: \"kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243931 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243904    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.243983 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243950    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.244024 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243977    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.244024 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.243897    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.245884 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.245865    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.251023 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.250997    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rdg\" (UniqueName: \"kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.376744 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.376655    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:00.548484 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.548461    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:35:00.550561 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:35:00.550531    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000fa7ca_8657_44d9_956d_68db6c614109.slice/crio-d327811d5c662aca83b0cd47afe2db985b8191be25f69164c8633c95b74661f7 WatchSource:0}: Error finding container d327811d5c662aca83b0cd47afe2db985b8191be25f69164c8633c95b74661f7: Status 404 returned error can't find the container with id d327811d5c662aca83b0cd47afe2db985b8191be25f69164c8633c95b74661f7
Jun 02 16:35:00.552399 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.552382    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:35:00.902546 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.902512    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerStarted","Data":"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"}
Jun 02 16:35:00.902546 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:00.902548    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerStarted","Data":"d327811d5c662aca83b0cd47afe2db985b8191be25f69164c8633c95b74661f7"}
Jun 02 16:35:01.906927 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:01.906890    2568 generic.go:358] "Generic (PLEG): container finished" podID="000fa7ca-8657-44d9-956d-68db6c614109" containerID="f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5" exitCode=0
Jun 02 16:35:01.907500 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:01.906974    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerDied","Data":"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"}
Jun 02 16:35:02.911980 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:02.911942    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerStarted","Data":"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"}
Jun 02 16:35:02.911980 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:02.911983    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerStarted","Data":"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"}
Jun 02 16:35:02.912451 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:02.912177    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:08.499567 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.499503    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" podStartSLOduration=8.499485065 podStartE2EDuration="8.499485065s" podCreationTimestamp="2026-06-02 16:35:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:35:02.950258427 +0000 UTC m=+2888.634806363" watchObservedRunningTime="2026-06-02 16:35:08.499485065 +0000 UTC m=+2894.184032999"
Jun 02 16:35:08.500081 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.499673    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"]
Jun 02 16:35:08.503176 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.503152    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.514951 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.514919    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"]
Jun 02 16:35:08.608989 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.608943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94b238f0-5cab-4f2f-96d3-df407f18629c-cert\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.609200 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.609009    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n59\" (UniqueName: \"kubernetes.io/projected/94b238f0-5cab-4f2f-96d3-df407f18629c-kube-api-access-h9n59\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.709509 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.709468    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94b238f0-5cab-4f2f-96d3-df407f18629c-cert\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.709671 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.709532    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n59\" (UniqueName: \"kubernetes.io/projected/94b238f0-5cab-4f2f-96d3-df407f18629c-kube-api-access-h9n59\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.711970 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.711941    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94b238f0-5cab-4f2f-96d3-df407f18629c-cert\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.718419 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.718385    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n59\" (UniqueName: \"kubernetes.io/projected/94b238f0-5cab-4f2f-96d3-df407f18629c-kube-api-access-h9n59\") pod \"llmisvc-controller-manager-86c7bf9457-dxv9d\" (UID: \"94b238f0-5cab-4f2f-96d3-df407f18629c\") " pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.814739 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.814640    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:08.946562 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:08.946540    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"]
Jun 02 16:35:08.949089 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:35:08.949036    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94b238f0_5cab_4f2f_96d3_df407f18629c.slice/crio-f977a1f9e434e3b38655685f03bca499ce102bb2238594e0147ac1f73c9e8f65 WatchSource:0}: Error finding container f977a1f9e434e3b38655685f03bca499ce102bb2238594e0147ac1f73c9e8f65: Status 404 returned error can't find the container with id f977a1f9e434e3b38655685f03bca499ce102bb2238594e0147ac1f73c9e8f65
Jun 02 16:35:09.938069 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:09.938020    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d" event={"ID":"94b238f0-5cab-4f2f-96d3-df407f18629c","Type":"ContainerStarted","Data":"eda018c7512615c78662e0d1df1a030192bd2d4a8ffc8ef46ad69738b9a6d1f4"}
Jun 02 16:35:09.938451 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:09.938077    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d" event={"ID":"94b238f0-5cab-4f2f-96d3-df407f18629c","Type":"ContainerStarted","Data":"f977a1f9e434e3b38655685f03bca499ce102bb2238594e0147ac1f73c9e8f65"}
Jun 02 16:35:09.938451 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:09.938174    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:09.956153 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:09.956095    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d" podStartSLOduration=1.4049329959999999 podStartE2EDuration="1.95607868s" podCreationTimestamp="2026-06-02 16:35:08 +0000 UTC" firstStartedPulling="2026-06-02 16:35:08.950437851 +0000 UTC m=+2894.634985779" lastFinishedPulling="2026-06-02 16:35:09.50158355 +0000 UTC m=+2895.186131463" observedRunningTime="2026-06-02 16:35:09.954375826 +0000 UTC m=+2895.638923762" watchObservedRunningTime="2026-06-02 16:35:09.95607868 +0000 UTC m=+2895.640626613"
Jun 02 16:35:10.377402 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:10.377311    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:10.377402 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:10.377360    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:10.379938 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:10.379915    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:10.942393 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:10.942362    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:31.945967 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:31.945937    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:35:40.943627 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:40.943597    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-86c7bf9457-dxv9d"
Jun 02 16:35:40.986574 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:40.986536    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 16:35:40.986818 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:40.986792    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" podUID="326ee9ff-a6ba-4e5d-9370-357e6270c463" containerName="manager" containerID="cri-o://7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a" gracePeriod=30
Jun 02 16:35:41.229625 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.229601    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 16:35:41.379414 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.379377    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert\") pod \"326ee9ff-a6ba-4e5d-9370-357e6270c463\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") "
Jun 02 16:35:41.379592 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.379430    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqqd7\" (UniqueName: \"kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7\") pod \"326ee9ff-a6ba-4e5d-9370-357e6270c463\" (UID: \"326ee9ff-a6ba-4e5d-9370-357e6270c463\") "
Jun 02 16:35:41.381505 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.381466    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7" (OuterVolumeSpecName: "kube-api-access-pqqd7") pod "326ee9ff-a6ba-4e5d-9370-357e6270c463" (UID: "326ee9ff-a6ba-4e5d-9370-357e6270c463"). InnerVolumeSpecName "kube-api-access-pqqd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:35:41.381505 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.381491    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert" (OuterVolumeSpecName: "cert") pod "326ee9ff-a6ba-4e5d-9370-357e6270c463" (UID: "326ee9ff-a6ba-4e5d-9370-357e6270c463"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:35:41.480232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.480151    2568 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/326ee9ff-a6ba-4e5d-9370-357e6270c463-cert\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:35:41.480232 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:41.480181    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqd7\" (UniqueName: \"kubernetes.io/projected/326ee9ff-a6ba-4e5d-9370-357e6270c463-kube-api-access-pqqd7\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:35:42.041428 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.041397    2568 generic.go:358] "Generic (PLEG): container finished" podID="326ee9ff-a6ba-4e5d-9370-357e6270c463" containerID="7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a" exitCode=0
Jun 02 16:35:42.041817 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.041435    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" event={"ID":"326ee9ff-a6ba-4e5d-9370-357e6270c463","Type":"ContainerDied","Data":"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"}
Jun 02 16:35:42.041817 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.041459    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh" event={"ID":"326ee9ff-a6ba-4e5d-9370-357e6270c463","Type":"ContainerDied","Data":"42c69bb0b6e1226132cffdcb1587baca74b7eec7002202d574f2bcabc76f3b69"}
Jun 02 16:35:42.041817 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.041463    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-74846f4db6-jx2gh"
Jun 02 16:35:42.041817 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.041473    2568 scope.go:117] "RemoveContainer" containerID="7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"
Jun 02 16:35:42.049931 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.049913    2568 scope.go:117] "RemoveContainer" containerID="7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"
Jun 02 16:35:42.050208 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:35:42.050187    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a\": container with ID starting with 7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a not found: ID does not exist" containerID="7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"
Jun 02 16:35:42.050259 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.050217    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a"} err="failed to get container status \"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a\": rpc error: code = NotFound desc = could not find container \"7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a\": container with ID starting with 7e3b0ac3b213bf1c27109abdd029a78b5038ffae4d7e7b9002673ebb305d283a not found: ID does not exist"
Jun 02 16:35:42.061924 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.061900    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 16:35:42.065597 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.065576    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-74846f4db6-jx2gh"]
Jun 02 16:35:42.892932 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:35:42.892896    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326ee9ff-a6ba-4e5d-9370-357e6270c463" path="/var/lib/kubelet/pods/326ee9ff-a6ba-4e5d-9370-357e6270c463/volumes"
Jun 02 16:36:54.972482 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:36:54.972378    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:36:54.980508 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:36:54.980489    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:41:54.993771 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:41:54.993647    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:41:55.001729 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:41:55.001709    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:46:55.014259 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:46:55.014142    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:46:55.023082 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:46:55.023043    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:49:55.014734 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:55.014700    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:49:55.015360 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:55.014997    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="main" containerID="cri-o://4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7" gracePeriod=30
Jun 02 16:49:55.015360 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:55.015069    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="tokenizer" containerID="cri-o://034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc" gracePeriod=30
Jun 02 16:49:55.688935 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:55.688903    2568 generic.go:358] "Generic (PLEG): container finished" podID="000fa7ca-8657-44d9-956d-68db6c614109" containerID="4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7" exitCode=0
Jun 02 16:49:55.689166 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:55.688974    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerDied","Data":"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"}
Jun 02 16:49:56.267698 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.267673    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:49:56.417124 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417081    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417130    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8rdg\" (UniqueName: \"kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417174    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417211    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417255    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417298 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417289    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location\") pod \"000fa7ca-8657-44d9-956d-68db6c614109\" (UID: \"000fa7ca-8657-44d9-956d-68db6c614109\") "
Jun 02 16:49:56.417572 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417368    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:49:56.417572 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417495    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:49:56.417662 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417595    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.417662 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417615    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.417662 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417617    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:49:56.418011 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.417991    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:49:56.419355 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.419332    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:49:56.419355 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.419346    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg" (OuterVolumeSpecName: "kube-api-access-x8rdg") pod "000fa7ca-8657-44d9-956d-68db6c614109" (UID: "000fa7ca-8657-44d9-956d-68db6c614109"). InnerVolumeSpecName "kube-api-access-x8rdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:49:56.518932 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.518899    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.518932 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.518928    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000fa7ca-8657-44d9-956d-68db6c614109-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.518932 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.518940    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rdg\" (UniqueName: \"kubernetes.io/projected/000fa7ca-8657-44d9-956d-68db6c614109-kube-api-access-x8rdg\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.519187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.518951    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000fa7ca-8657-44d9-956d-68db6c614109-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:49:56.694391 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.694302    2568 generic.go:358] "Generic (PLEG): container finished" podID="000fa7ca-8657-44d9-956d-68db6c614109" containerID="034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc" exitCode=0
Jun 02 16:49:56.694520 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.694402    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"
Jun 02 16:49:56.694520 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.694399    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerDied","Data":"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"}
Jun 02 16:49:56.694597 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.694526    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc" event={"ID":"000fa7ca-8657-44d9-956d-68db6c614109","Type":"ContainerDied","Data":"d327811d5c662aca83b0cd47afe2db985b8191be25f69164c8633c95b74661f7"}
Jun 02 16:49:56.694597 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.694553    2568 scope.go:117] "RemoveContainer" containerID="034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"
Jun 02 16:49:56.702356 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.702338    2568 scope.go:117] "RemoveContainer" containerID="4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"
Jun 02 16:49:56.709252 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.709234    2568 scope.go:117] "RemoveContainer" containerID="f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"
Jun 02 16:49:56.714998 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.714977    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:49:56.718552 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.718533    2568 scope.go:117] "RemoveContainer" containerID="034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"
Jun 02 16:49:56.718821 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:49:56.718791    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc\": container with ID starting with 034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc not found: ID does not exist" containerID="034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"
Jun 02 16:49:56.718865 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.718831    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc"} err="failed to get container status \"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc\": rpc error: code = NotFound desc = could not find container \"034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc\": container with ID starting with 034c93d79d9db9e40e78c55069b79e5711a06d4b8e814301c613f89c1ec6b2dc not found: ID does not exist"
Jun 02 16:49:56.718865 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.718847    2568 scope.go:117] "RemoveContainer" containerID="4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"
Jun 02 16:49:56.719158 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:49:56.719130    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7\": container with ID starting with 4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7 not found: ID does not exist" containerID="4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"
Jun 02 16:49:56.719246 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.719167    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7"} err="failed to get container status \"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7\": rpc error: code = NotFound desc = could not find container \"4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7\": container with ID starting with 4c0c24bdefce5ace8c06069dc461371b4e3a84e6ee86fa8c09c0b974aa135ea7 not found: ID does not exist"
Jun 02 16:49:56.719246 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.719227    2568 scope.go:117] "RemoveContainer" containerID="f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"
Jun 02 16:49:56.719921 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:49:56.719665    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5\": container with ID starting with f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5 not found: ID does not exist" containerID="f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"
Jun 02 16:49:56.719921 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.719699    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5"} err="failed to get container status \"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5\": rpc error: code = NotFound desc = could not find container \"f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5\": container with ID starting with f80a2c53c6e68431cc23cdba697962b4fdba3689447bbf0508bf307245c0b4f5 not found: ID does not exist"
Jun 02 16:49:56.725349 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.723500    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche7vqhc"]
Jun 02 16:49:56.893409 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:49:56.893378    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000fa7ca-8657-44d9-956d-68db6c614109" path="/var/lib/kubelet/pods/000fa7ca-8657-44d9-956d-68db6c614109/volumes"
Jun 02 16:50:02.573099 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573063    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573329    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326ee9ff-a6ba-4e5d-9370-357e6270c463" containerName="manager"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573339    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="326ee9ff-a6ba-4e5d-9370-357e6270c463" containerName="manager"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573350    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="main"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573357    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="main"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573373    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="tokenizer"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573378    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="tokenizer"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573386    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="storage-initializer"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573391    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="storage-initializer"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573445    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="main"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573456    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="000fa7ca-8657-44d9-956d-68db6c614109" containerName="tokenizer"
Jun 02 16:50:02.573583 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.573462    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="326ee9ff-a6ba-4e5d-9370-357e6270c463" containerName="manager"
Jun 02 16:50:02.578559 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.578536    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.580870 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.580843    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:50:02.580995 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.580874    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jun 02 16:50:02.581421 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.581393    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-flzv6\""
Jun 02 16:50:02.581539 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.581454    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:50:02.581539 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.581485    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:50:02.588530 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.588511    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 16:50:02.664509 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.664649 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664517    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.664649 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.664649 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664620    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk9f\" (UniqueName: \"kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.664759 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.664759 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.664730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.765942 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.765909    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.765942 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.765947    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.765972    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766000    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766032    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk9f\" (UniqueName: \"kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766180 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766080    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766410 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766383    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766474 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766440    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766510 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766484    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.766510 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.766504    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.768448 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.768428    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.774165 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.774139    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk9f\" (UniqueName: \"kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:02.890203 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:02.890166    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:03.019385 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:03.019340    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 16:50:03.021823 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:50:03.021792    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5879ba8_8a04_42d6_b09f_241e7ce83e6e.slice/crio-4a0ba63b57ef9a8effa7c5e49fbfb0dc7d4ece4596817c9acbb90c4b3c5f3d2f WatchSource:0}: Error finding container 4a0ba63b57ef9a8effa7c5e49fbfb0dc7d4ece4596817c9acbb90c4b3c5f3d2f: Status 404 returned error can't find the container with id 4a0ba63b57ef9a8effa7c5e49fbfb0dc7d4ece4596817c9acbb90c4b3c5f3d2f
Jun 02 16:50:03.023629 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:03.023612    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:50:03.719603 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:03.719576    2568 generic.go:358] "Generic (PLEG): container finished" podID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerID="e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd" exitCode=0
Jun 02 16:50:03.719918 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:03.719644    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerDied","Data":"e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd"}
Jun 02 16:50:03.719918 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:03.719667    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerStarted","Data":"4a0ba63b57ef9a8effa7c5e49fbfb0dc7d4ece4596817c9acbb90c4b3c5f3d2f"}
Jun 02 16:50:04.725547 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:04.725510    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerStarted","Data":"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"}
Jun 02 16:50:04.725547 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:04.725550    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerStarted","Data":"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"}
Jun 02 16:50:04.725945 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:04.725645    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:04.746369 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:04.746326    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" podStartSLOduration=2.746311386 podStartE2EDuration="2.746311386s" podCreationTimestamp="2026-06-02 16:50:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:50:04.74416882 +0000 UTC m=+3790.428716761" watchObservedRunningTime="2026-06-02 16:50:04.746311386 +0000 UTC m=+3790.430859320"
Jun 02 16:50:12.893291 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:12.893265    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:12.893669 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:12.893386    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:12.893669 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:12.893399    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:12.894564 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:12.894540    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:50:34.761059 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:50:34.761027    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 16:51:24.147792 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.147709    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:51:24.150931 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.150914    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.153061 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.153030    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-pf4zd\""
Jun 02 16:51:24.153619 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.153604    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jun 02 16:51:24.156638 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.156614    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:51:24.217897 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.217860    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218077 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.217905    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218077 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.217968    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218077 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.218014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218223 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.218092    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218223 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.218111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pwr\" (UniqueName: \"kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.218223 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.218132    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.318968 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.318931    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.318979    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319014    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319100    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319126    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pwr\" (UniqueName: \"kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319187 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319160    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319197    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319342    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319400    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319473 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319459    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.319611 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.319508    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.321333 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.321308    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.321493 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.321476    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.325949 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.325929    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pwr\" (UniqueName: \"kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.462739 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.462663    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:51:24.591337 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.591153    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:51:24.593913 ip-10-0-135-190 kubenswrapper[2568]: W0602 16:51:24.593885    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a18a71c_c8f6_44b3_a996_f0966e08bb6c.slice/crio-f948a5d6d72fde4dc52d2fc8298f679cbcd0e6ffd3869e97cff17639dfe1f02a WatchSource:0}: Error finding container f948a5d6d72fde4dc52d2fc8298f679cbcd0e6ffd3869e97cff17639dfe1f02a: Status 404 returned error can't find the container with id f948a5d6d72fde4dc52d2fc8298f679cbcd0e6ffd3869e97cff17639dfe1f02a
Jun 02 16:51:24.971394 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.971355    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6a18a71c-c8f6-44b3-a996-f0966e08bb6c","Type":"ContainerStarted","Data":"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"}
Jun 02 16:51:24.971394 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:24.971397    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6a18a71c-c8f6-44b3-a996-f0966e08bb6c","Type":"ContainerStarted","Data":"f948a5d6d72fde4dc52d2fc8298f679cbcd0e6ffd3869e97cff17639dfe1f02a"}
Jun 02 16:51:55.035757 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:55.035652    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:51:55.045203 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:51:55.045180    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:54:04.282484 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:04.282448    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:54:04.283032 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:04.282718    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" containerName="storage-initializer" containerID="cri-o://72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b" gracePeriod=30
Jun 02 16:54:34.478328 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.478303    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_6a18a71c-c8f6-44b3-a996-f0966e08bb6c/storage-initializer/0.log"
Jun 02 16:54:34.478640 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.478367    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:54:34.570218 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570129    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570218 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570171    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570218 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570214    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570282    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570321    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570353    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570506 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570394    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9pwr\" (UniqueName: \"kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr\") pod \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\" (UID: \"6a18a71c-c8f6-44b3-a996-f0966e08bb6c\") "
Jun 02 16:54:34.570723 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570516    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home" (OuterVolumeSpecName: "home") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:34.570723 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570523    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:34.570723 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.570617    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:34.572599 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.572562    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:54:34.572789 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.572765    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm" (OuterVolumeSpecName: "dshm") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:34.572890 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.572834    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr" (OuterVolumeSpecName: "kube-api-access-t9pwr") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "kube-api-access-t9pwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:54:34.574627 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574611    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_6a18a71c-c8f6-44b3-a996-f0966e08bb6c/storage-initializer/0.log"
Jun 02 16:54:34.574734 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574645    2568 generic.go:358] "Generic (PLEG): container finished" podID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" containerID="72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b" exitCode=137
Jun 02 16:54:34.574802 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574734    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
Jun 02 16:54:34.574802 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574735    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6a18a71c-c8f6-44b3-a996-f0966e08bb6c","Type":"ContainerDied","Data":"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"}
Jun 02 16:54:34.574802 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574784    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6a18a71c-c8f6-44b3-a996-f0966e08bb6c","Type":"ContainerDied","Data":"f948a5d6d72fde4dc52d2fc8298f679cbcd0e6ffd3869e97cff17639dfe1f02a"}
Jun 02 16:54:34.574949 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.574804    2568 scope.go:117] "RemoveContainer" containerID="72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"
Jun 02 16:54:34.589721 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.589695    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a18a71c-c8f6-44b3-a996-f0966e08bb6c" (UID: "6a18a71c-c8f6-44b3-a996-f0966e08bb6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:34.607833 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.607816    2568 scope.go:117] "RemoveContainer" containerID="72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"
Jun 02 16:54:34.608161 ip-10-0-135-190 kubenswrapper[2568]: E0602 16:54:34.608142    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b\": container with ID starting with 72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b not found: ID does not exist" containerID="72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"
Jun 02 16:54:34.608229 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.608170    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b"} err="failed to get container status \"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b\": rpc error: code = NotFound desc = could not find container \"72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b\": container with ID starting with 72a4b6d5b5aeff8fac97daf6da68b528b15cb3a6095ff250a570d7739683005b not found: ID does not exist"
Jun 02 16:54:34.671358 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671319    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-model-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671358 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671351    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671358 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671365    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9pwr\" (UniqueName: \"kubernetes.io/projected/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kube-api-access-t9pwr\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671602 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671378    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-dshm\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671602 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671393    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671602 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671407    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-home\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.671602 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.671419    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a18a71c-c8f6-44b3-a996-f0966e08bb6c-tmp-dir\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 16:54:34.909748 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.909689    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:54:34.912069 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:34.912030    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
Jun 02 16:54:36.893385 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:54:36.893356    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" path="/var/lib/kubelet/pods/6a18a71c-c8f6-44b3-a996-f0966e08bb6c/volumes"
Jun 02 16:56:55.056067 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:56:55.055940    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 16:56:55.066022 ip-10-0-135-190 kubenswrapper[2568]: I0602 16:56:55.066000    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:01:55.077069 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:01:55.076943    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:01:55.087754 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:01:55.087735    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:04:59.096169 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.096132    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 17:04:59.096816 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.096531    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="main" containerID="cri-o://a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde" gracePeriod=30
Jun 02 17:04:59.096816 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.096613    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="tokenizer" containerID="cri-o://95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04" gracePeriod=30
Jun 02 17:04:59.387190 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.387150    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"]
Jun 02 17:04:59.387588 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.387571    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" containerName="storage-initializer"
Jun 02 17:04:59.387658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.387591    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" containerName="storage-initializer"
Jun 02 17:04:59.387699 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.387685    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a18a71c-c8f6-44b3-a996-f0966e08bb6c" containerName="storage-initializer"
Jun 02 17:04:59.390723 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.390705    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.393810 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.393787    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-xmvhb\""
Jun 02 17:04:59.394449 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.394427    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jun 02 17:04:59.416371 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.416341    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"]
Jun 02 17:04:59.506508 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.506479    2568 generic.go:358] "Generic (PLEG): container finished" podID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerID="a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde" exitCode=0
Jun 02 17:04:59.506676 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.506514    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerDied","Data":"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"}
Jun 02 17:04:59.537886 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.537853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.537886 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.537892    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnntn\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-kube-api-access-xnntn\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538134 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.537922    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9372b7f-1b4b-4131-b845-e3e11a336faa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538134 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.537995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538134 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.538021    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538134 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.538066    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538134 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.538084    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538345 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.538148    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.538345 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.538169    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639485 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639379    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639485 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639426    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639485 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639460    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639487    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639539    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639566    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xnntn\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-kube-api-access-xnntn\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639608    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9372b7f-1b4b-4131-b845-e3e11a336faa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639645    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.639811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639672    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.640163 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.639998    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.640163 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.640023    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.640270 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.640185    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.640270 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.640213    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.640510 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.640488    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9372b7f-1b4b-4131-b845-e3e11a336faa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.641939 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.641915    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.642317 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.642301    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.648430 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.648402    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.648527 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.648467    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnntn\" (UniqueName: \"kubernetes.io/projected/d9372b7f-1b4b-4131-b845-e3e11a336faa-kube-api-access-xnntn\") pod \"router-gateway-2-openshift-default-7bb57db77b-2hl9r\" (UID: \"d9372b7f-1b4b-4131-b845-e3e11a336faa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.703310 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.703269    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:04:59.847248 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.847215    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"]
Jun 02 17:04:59.851022 ip-10-0-135-190 kubenswrapper[2568]: W0602 17:04:59.850995    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9372b7f_1b4b_4131_b845_e3e11a336faa.slice/crio-24252ccf5e2bf465a32d4e25129f026cd4fcf920d3ebb54f01e0aa3a27010f46 WatchSource:0}: Error finding container 24252ccf5e2bf465a32d4e25129f026cd4fcf920d3ebb54f01e0aa3a27010f46: Status 404 returned error can't find the container with id 24252ccf5e2bf465a32d4e25129f026cd4fcf920d3ebb54f01e0aa3a27010f46
Jun 02 17:04:59.852819 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:04:59.852806    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 17:05:00.347008 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.346986    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 17:05:00.447672 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447594    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xk9f\" (UniqueName: \"kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.447672 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447669    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.447897 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447689    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.447897 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447720    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.447897 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447764    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.447897 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.447788    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds\") pod \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\" (UID: \"a5879ba8-8a04-42d6-b09f-241e7ce83e6e\") "
Jun 02 17:05:00.448127 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.448041    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:00.448127 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.448084    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:00.448203 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.448135    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:00.448497 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.448477    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:00.449811 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.449789    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f" (OuterVolumeSpecName: "kube-api-access-8xk9f") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "kube-api-access-8xk9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:05:00.449864 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.449807    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5879ba8-8a04-42d6-b09f-241e7ce83e6e" (UID: "a5879ba8-8a04-42d6-b09f-241e7ce83e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:05:00.510844 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.510804    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" event={"ID":"d9372b7f-1b4b-4131-b845-e3e11a336faa","Type":"ContainerStarted","Data":"24252ccf5e2bf465a32d4e25129f026cd4fcf920d3ebb54f01e0aa3a27010f46"}
Jun 02 17:05:00.512271 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.512247    2568 generic.go:358] "Generic (PLEG): container finished" podID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerID="95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04" exitCode=0
Jun 02 17:05:00.512371 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.512305    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerDied","Data":"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"}
Jun 02 17:05:00.512371 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.512326    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"
Jun 02 17:05:00.512371 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.512344    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8" event={"ID":"a5879ba8-8a04-42d6-b09f-241e7ce83e6e","Type":"ContainerDied","Data":"4a0ba63b57ef9a8effa7c5e49fbfb0dc7d4ece4596817c9acbb90c4b3c5f3d2f"}
Jun 02 17:05:00.512371 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.512362    2568 scope.go:117] "RemoveContainer" containerID="95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"
Jun 02 17:05:00.520865 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.520849    2568 scope.go:117] "RemoveContainer" containerID="a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"
Jun 02 17:05:00.527769 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.527750    2568 scope.go:117] "RemoveContainer" containerID="e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd"
Jun 02 17:05:00.534893 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.534873    2568 scope.go:117] "RemoveContainer" containerID="95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"
Jun 02 17:05:00.535269 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:05:00.535235    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04\": container with ID starting with 95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04 not found: ID does not exist" containerID="95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"
Jun 02 17:05:00.535380 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.535268    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04"} err="failed to get container status \"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04\": rpc error: code = NotFound desc = could not find container \"95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04\": container with ID starting with 95062892ddf3f0debf0b0986501c651f6f038c68742f61892dbb5ef4b4c6ad04 not found: ID does not exist"
Jun 02 17:05:00.535380 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.535293    2568 scope.go:117] "RemoveContainer" containerID="a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"
Jun 02 17:05:00.535564 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:05:00.535544    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde\": container with ID starting with a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde not found: ID does not exist" containerID="a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"
Jun 02 17:05:00.535618 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.535570    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde"} err="failed to get container status \"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde\": rpc error: code = NotFound desc = could not find container \"a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde\": container with ID starting with a79d8c5d9ffe580d6f1c9275a98acf95b9b1c0bda31716fb63d55159b91d3dde not found: ID does not exist"
Jun 02 17:05:00.535618 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.535585    2568 scope.go:117] "RemoveContainer" containerID="e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd"
Jun 02 17:05:00.535866 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:05:00.535850    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd\": container with ID starting with e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd not found: ID does not exist" containerID="e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd"
Jun 02 17:05:00.535927 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.535870    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd"} err="failed to get container status \"e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd\": rpc error: code = NotFound desc = could not find container \"e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd\": container with ID starting with e1880b1f7f1674d4e0e0ce80811ab570698143e7507c06f3a22508c9ae6c2cbd not found: ID does not exist"
Jun 02 17:05:00.536588 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.536568    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 17:05:00.542868 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.542846    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-7d7b7j65f8"]
Jun 02 17:05:00.549224 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549207    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.549301 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549226    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk9f\" (UniqueName: \"kubernetes.io/projected/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kube-api-access-8xk9f\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.549301 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549237    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.549301 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549247    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.549301 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549255    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.549301 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.549263    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5879ba8-8a04-42d6-b09f-241e7ce83e6e-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:05:00.894321 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:00.894281    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" path="/var/lib/kubelet/pods/a5879ba8-8a04-42d6-b09f-241e7ce83e6e/volumes"
Jun 02 17:05:02.431349 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.431315    2568 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jun 02 17:05:02.431626 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.431381    2568 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jun 02 17:05:02.431626 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.431407    2568 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jun 02 17:05:02.520922 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.520895    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" event={"ID":"d9372b7f-1b4b-4131-b845-e3e11a336faa","Type":"ContainerStarted","Data":"e3a6ab4563fd2c12cf3895ac6642af23e4e0934f92be5c9ffd1509b6fa50d551"}
Jun 02 17:05:02.541708 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.541643    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" podStartSLOduration=0.963472373 podStartE2EDuration="3.541622972s" podCreationTimestamp="2026-06-02 17:04:59 +0000 UTC" firstStartedPulling="2026-06-02 17:04:59.85293502 +0000 UTC m=+4685.537482932" lastFinishedPulling="2026-06-02 17:05:02.431085606 +0000 UTC m=+4688.115633531" observedRunningTime="2026-06-02 17:05:02.539479501 +0000 UTC m=+4688.224027436" watchObservedRunningTime="2026-06-02 17:05:02.541622972 +0000 UTC m=+4688.226170913"
Jun 02 17:05:02.703851 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:02.703780    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:05:03.704911 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:03.704829    2568 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" podUID="d9372b7f-1b4b-4131-b845-e3e11a336faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jun 02 17:05:04.704485 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:04.704444    2568 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" podUID="d9372b7f-1b4b-4131-b845-e3e11a336faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jun 02 17:05:05.703801 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:05.703759    2568 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r" podUID="d9372b7f-1b4b-4131-b845-e3e11a336faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jun 02 17:05:06.708018 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:06.707986    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:05:06.708415 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:06.708280    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:05:06.709034 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:06.709016    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-2hl9r"
Jun 02 17:05:09.779539 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779504    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779815    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="main"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779826    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="main"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779833    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="tokenizer"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779839    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="tokenizer"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779849    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="storage-initializer"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779855    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="storage-initializer"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779910    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="tokenizer"
Jun 02 17:05:09.779924 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.779917    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="a5879ba8-8a04-42d6-b09f-241e7ce83e6e" containerName="main"
Jun 02 17:05:09.783851 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.783831    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.786333 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.786309    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 17:05:09.786706 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.786678    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-72vpd\""
Jun 02 17:05:09.786823 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.786678    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jun 02 17:05:09.792927 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.792902    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:05:09.931718 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931685    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.931895 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.931895 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.931895 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931805    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpjp\" (UniqueName: \"kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.931895 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:09.931895 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:09.931894    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032611 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032515    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpjp\" (UniqueName: \"kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032611 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032580    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032611 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032601    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032922 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032702    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032922 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032756    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.032922 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.032786    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.033286 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.033255    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.033433 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.033404    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.033517 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.033499    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.033576 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.033524    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.035413 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.035393    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.043621 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.043596    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpjp\" (UniqueName: \"kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.094446 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.094411    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:10.225162 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.225119    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:05:10.228238 ip-10-0-135-190 kubenswrapper[2568]: W0602 17:05:10.228209    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceda9d1c_d360_4a5b_b34a_bb250205caa4.slice/crio-a135829fa0713c5bee11434d688662bb83a17264fbd28303ca170ed8ef3fe29c WatchSource:0}: Error finding container a135829fa0713c5bee11434d688662bb83a17264fbd28303ca170ed8ef3fe29c: Status 404 returned error can't find the container with id a135829fa0713c5bee11434d688662bb83a17264fbd28303ca170ed8ef3fe29c
Jun 02 17:05:10.549347 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.549309    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerStarted","Data":"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"}
Jun 02 17:05:10.549347 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:10.549349    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerStarted","Data":"a135829fa0713c5bee11434d688662bb83a17264fbd28303ca170ed8ef3fe29c"}
Jun 02 17:05:11.554011 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:11.553973    2568 generic.go:358] "Generic (PLEG): container finished" podID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerID="f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540" exitCode=0
Jun 02 17:05:11.554439 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:11.554078    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerDied","Data":"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"}
Jun 02 17:05:12.560482 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:12.560448    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerStarted","Data":"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"}
Jun 02 17:05:12.560482 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:12.560485    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerStarted","Data":"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"}
Jun 02 17:05:12.560925 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:12.560591    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:12.617818 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:12.617753    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" podStartSLOduration=3.617732239 podStartE2EDuration="3.617732239s" podCreationTimestamp="2026-06-02 17:05:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:05:12.614903824 +0000 UTC m=+4698.299451759" watchObservedRunningTime="2026-06-02 17:05:12.617732239 +0000 UTC m=+4698.302280174"
Jun 02 17:05:20.095014 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:20.094971    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:20.095514 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:20.095029    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:20.097757 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:20.097733    2568 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:20.594235 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:20.594210    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:05:41.598515 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:05:41.598485    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:06:55.098691 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:06:55.098583    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:06:55.109440 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:06:55.109416    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:07:42.026109 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:42.026014    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:07:42.026604 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:42.026452    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="main" containerID="cri-o://af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765" gracePeriod=30
Jun 02 17:07:42.026604 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:42.026494    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="tokenizer" containerID="cri-o://ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73" gracePeriod=30
Jun 02 17:07:43.039402 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.039374    2568 generic.go:358] "Generic (PLEG): container finished" podID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerID="af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765" exitCode=0
Jun 02 17:07:43.039739 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.039457    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerDied","Data":"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"}
Jun 02 17:07:43.176767 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.176742    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:07:43.306172 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306090    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306172 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306125    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbpjp\" (UniqueName: \"kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306172 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306152    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306438 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306182    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306438 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306218    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306438 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306313    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location\") pod \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\" (UID: \"ceda9d1c-d360-4a5b-b34a-bb250205caa4\") "
Jun 02 17:07:43.306596 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306488    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:07:43.306596 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306564    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:07:43.306672 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.306605    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:07:43.307242 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.307211    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:07:43.308190 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.308171    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:07:43.308292 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.308240    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp" (OuterVolumeSpecName: "kube-api-access-gbpjp") pod "ceda9d1c-d360-4a5b-b34a-bb250205caa4" (UID: "ceda9d1c-d360-4a5b-b34a-bb250205caa4"). InnerVolumeSpecName "kube-api-access-gbpjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:07:43.407899 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407865    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kserve-provision-location\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.407899 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407893    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tls-certs\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.407899 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407904    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpjp\" (UniqueName: \"kubernetes.io/projected/ceda9d1c-d360-4a5b-b34a-bb250205caa4-kube-api-access-gbpjp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.408143 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407914    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-uds\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.408143 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407923    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-tmp\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.408143 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.407931    2568 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceda9d1c-d360-4a5b-b34a-bb250205caa4-tokenizer-cache\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:07:43.612409 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612327    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bj4rc/must-gather-jdd4w"]
Jun 02 17:07:43.612612 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612601    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="tokenizer"
Jun 02 17:07:43.612658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612614    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="tokenizer"
Jun 02 17:07:43.612658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612628    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="main"
Jun 02 17:07:43.612658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612635    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="main"
Jun 02 17:07:43.612658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612644    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="storage-initializer"
Jun 02 17:07:43.612658 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612650    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="storage-initializer"
Jun 02 17:07:43.612805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612701    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="main"
Jun 02 17:07:43.612805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.612713    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerName="tokenizer"
Jun 02 17:07:43.616668 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.616647    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.619522 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.619499    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bj4rc\"/\"kube-root-ca.crt\""
Jun 02 17:07:43.619645 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.619556    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bj4rc\"/\"openshift-service-ca.crt\""
Jun 02 17:07:43.619796 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.619782    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bj4rc\"/\"default-dockercfg-h9jnd\""
Jun 02 17:07:43.629232 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.629210    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bj4rc/must-gather-jdd4w"]
Jun 02 17:07:43.710332 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.710293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2wg\" (UniqueName: \"kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.710520 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.710340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.810916 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.810876    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2wg\" (UniqueName: \"kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.810916 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.810924    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.811223 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.811208    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.819461 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.819435    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2wg\" (UniqueName: \"kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg\") pod \"must-gather-jdd4w\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") " pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:43.926406 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:43.926371    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:07:44.044233 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.044203    2568 generic.go:358] "Generic (PLEG): container finished" podID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" containerID="ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73" exitCode=0
Jun 02 17:07:44.044622 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.044271    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"
Jun 02 17:07:44.044622 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.044278    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerDied","Data":"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"}
Jun 02 17:07:44.044622 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.044314    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7" event={"ID":"ceda9d1c-d360-4a5b-b34a-bb250205caa4","Type":"ContainerDied","Data":"a135829fa0713c5bee11434d688662bb83a17264fbd28303ca170ed8ef3fe29c"}
Jun 02 17:07:44.044622 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.044330    2568 scope.go:117] "RemoveContainer" containerID="ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"
Jun 02 17:07:44.050738 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.050688    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bj4rc/must-gather-jdd4w"]
Jun 02 17:07:44.052678 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.052657    2568 scope.go:117] "RemoveContainer" containerID="af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"
Jun 02 17:07:44.054728 ip-10-0-135-190 kubenswrapper[2568]: W0602 17:07:44.054707    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b8ed89d_4f7f_4d63_ac0d_8c24cf18e778.slice/crio-9139723dd6fef304617aa2e3d92f4dbd053956445d51148881d33be3cf208338 WatchSource:0}: Error finding container 9139723dd6fef304617aa2e3d92f4dbd053956445d51148881d33be3cf208338: Status 404 returned error can't find the container with id 9139723dd6fef304617aa2e3d92f4dbd053956445d51148881d33be3cf208338
Jun 02 17:07:44.060492 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.060474    2568 scope.go:117] "RemoveContainer" containerID="f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"
Jun 02 17:07:44.067459 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.067445    2568 scope.go:117] "RemoveContainer" containerID="ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"
Jun 02 17:07:44.067694 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:07:44.067677    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73\": container with ID starting with ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73 not found: ID does not exist" containerID="ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"
Jun 02 17:07:44.067747 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.067702    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73"} err="failed to get container status \"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73\": rpc error: code = NotFound desc = could not find container \"ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73\": container with ID starting with ec3e70c9076f2cc0793077230fb222ab3144f4b86c61b5edbff248bcb427bc73 not found: ID does not exist"
Jun 02 17:07:44.067747 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.067721    2568 scope.go:117] "RemoveContainer" containerID="af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"
Jun 02 17:07:44.067929 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:07:44.067914    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765\": container with ID starting with af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765 not found: ID does not exist" containerID="af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"
Jun 02 17:07:44.067973 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.067932    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765"} err="failed to get container status \"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765\": rpc error: code = NotFound desc = could not find container \"af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765\": container with ID starting with af8a2086e518d0b366abe3f4c7c4e49194cd44c3fbbfc3939f9e9e7e4d3bf765 not found: ID does not exist"
Jun 02 17:07:44.067973 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.067946    2568 scope.go:117] "RemoveContainer" containerID="f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"
Jun 02 17:07:44.068205 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:07:44.068186    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540\": container with ID starting with f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540 not found: ID does not exist" containerID="f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"
Jun 02 17:07:44.068252 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.068214    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540"} err="failed to get container status \"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540\": rpc error: code = NotFound desc = could not find container \"f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540\": container with ID starting with f5264a1c863b64c7b5bdafa72f49869dae534528ac68b1ae0b2661606ed86540 not found: ID does not exist"
Jun 02 17:07:44.071451 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.071429    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:07:44.076938 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.076920    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cd969c89h85c7"]
Jun 02 17:07:44.895144 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:44.895107    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceda9d1c-d360-4a5b-b34a-bb250205caa4" path="/var/lib/kubelet/pods/ceda9d1c-d360-4a5b-b34a-bb250205caa4/volumes"
Jun 02 17:07:45.051242 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:45.051201    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" event={"ID":"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778","Type":"ContainerStarted","Data":"9139723dd6fef304617aa2e3d92f4dbd053956445d51148881d33be3cf208338"}
Jun 02 17:07:51.077388 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:51.077349    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" event={"ID":"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778","Type":"ContainerStarted","Data":"4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a"}
Jun 02 17:07:51.077388 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:51.077389    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" event={"ID":"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778","Type":"ContainerStarted","Data":"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"}
Jun 02 17:07:51.092723 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:51.092674    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" podStartSLOduration=2.061896367 podStartE2EDuration="8.092656304s" podCreationTimestamp="2026-06-02 17:07:43 +0000 UTC" firstStartedPulling="2026-06-02 17:07:44.056347377 +0000 UTC m=+4849.740895290" lastFinishedPulling="2026-06-02 17:07:50.087107315 +0000 UTC m=+4855.771655227" observedRunningTime="2026-06-02 17:07:51.091519891 +0000 UTC m=+4856.776067826" watchObservedRunningTime="2026-06-02 17:07:51.092656304 +0000 UTC m=+4856.777204241"
Jun 02 17:07:59.381997 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:07:59.381968    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:00.509122 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:00.509087    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:01.636235 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:01.636199    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:02.745730 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:02.745701    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:03.863594 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:03.863568    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:04.936076 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:04.936033    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:06.031892 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:06.031848    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:07.135259 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:07.135230    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:08.251390 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:08.251359    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:09.422673 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:09.422644    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:10.599851 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:10.599822    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:11.792966 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:11.792931    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:12.933918 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:12.933888    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:14.042068 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:14.042028    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-2hl9r_d9372b7f-1b4b-4131-b845-e3e11a336faa/istio-proxy/0.log"
Jun 02 17:08:17.561592 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:17.561564    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:17.574507 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:17.574485    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:18.467252 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:18.467224    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:18.480000 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:18.479977    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:19.397880 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:19.397813    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:19.411916 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:19.411895    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:20.300805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:20.300776    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:20.314617 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:20.314588    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:21.216332 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:21.216304    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:21.228716 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:21.228693    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:22.190373 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:22.190290    2568 generic.go:358] "Generic (PLEG): container finished" podID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerID="d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3" exitCode=0
Jun 02 17:08:22.190373 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:22.190362    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" event={"ID":"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778","Type":"ContainerDied","Data":"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"}
Jun 02 17:08:22.190692 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:22.190679    2568 scope.go:117] "RemoveContainer" containerID="d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"
Jun 02 17:08:22.934095 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:22.934069    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bj4rc_must-gather-jdd4w_4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778/gather/0.log"
Jun 02 17:08:23.708722 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.708690    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxgdw/must-gather-7ndwj"]
Jun 02 17:08:23.711651 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.711636    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.713957 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.713937    2568 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nxgdw\"/\"default-dockercfg-xs2rl\""
Jun 02 17:08:23.713957 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.713954    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxgdw\"/\"kube-root-ca.crt\""
Jun 02 17:08:23.714513 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.714497    2568 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxgdw\"/\"openshift-service-ca.crt\""
Jun 02 17:08:23.721377 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.721357    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/must-gather-7ndwj"]
Jun 02 17:08:23.758950 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.758921    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98a09f66-9bde-4726-8266-77cafc6c2e85-must-gather-output\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.759106 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.758970    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tl8\" (UniqueName: \"kubernetes.io/projected/98a09f66-9bde-4726-8266-77cafc6c2e85-kube-api-access-49tl8\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.859397 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.859360    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98a09f66-9bde-4726-8266-77cafc6c2e85-must-gather-output\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.859584 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.859441    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49tl8\" (UniqueName: \"kubernetes.io/projected/98a09f66-9bde-4726-8266-77cafc6c2e85-kube-api-access-49tl8\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.859704 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.859686    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98a09f66-9bde-4726-8266-77cafc6c2e85-must-gather-output\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:23.870380 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:23.870354    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tl8\" (UniqueName: \"kubernetes.io/projected/98a09f66-9bde-4726-8266-77cafc6c2e85-kube-api-access-49tl8\") pod \"must-gather-7ndwj\" (UID: \"98a09f66-9bde-4726-8266-77cafc6c2e85\") " pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:24.021292 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:24.021199    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/must-gather-7ndwj"
Jun 02 17:08:24.148620 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:24.148596    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/must-gather-7ndwj"]
Jun 02 17:08:24.151144 ip-10-0-135-190 kubenswrapper[2568]: W0602 17:08:24.151116    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a09f66_9bde_4726_8266_77cafc6c2e85.slice/crio-63c563e93f93ddb798f5961de2350632fdbacfad6494a1d0ecf427f9aff69cb0 WatchSource:0}: Error finding container 63c563e93f93ddb798f5961de2350632fdbacfad6494a1d0ecf427f9aff69cb0: Status 404 returned error can't find the container with id 63c563e93f93ddb798f5961de2350632fdbacfad6494a1d0ecf427f9aff69cb0
Jun 02 17:08:24.197628 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:24.197596    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/must-gather-7ndwj" event={"ID":"98a09f66-9bde-4726-8266-77cafc6c2e85","Type":"ContainerStarted","Data":"63c563e93f93ddb798f5961de2350632fdbacfad6494a1d0ecf427f9aff69cb0"}
Jun 02 17:08:25.202987 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:25.202943    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/must-gather-7ndwj" event={"ID":"98a09f66-9bde-4726-8266-77cafc6c2e85","Type":"ContainerStarted","Data":"fe66ce24575ceef43b587cd5a6e93cd0897185082e882abe4b2c1b1e04177173"}
Jun 02 17:08:26.209857 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:26.209817    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/must-gather-7ndwj" event={"ID":"98a09f66-9bde-4726-8266-77cafc6c2e85","Type":"ContainerStarted","Data":"3d6180b338ab047d195ac8897d0643e6d48f552a71877166a100a799ef0394a6"}
Jun 02 17:08:26.224212 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:26.224157    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nxgdw/must-gather-7ndwj" podStartSLOduration=2.299466281 podStartE2EDuration="3.224139677s" podCreationTimestamp="2026-06-02 17:08:23 +0000 UTC" firstStartedPulling="2026-06-02 17:08:24.152947535 +0000 UTC m=+4889.837495448" lastFinishedPulling="2026-06-02 17:08:25.077620916 +0000 UTC m=+4890.762168844" observedRunningTime="2026-06-02 17:08:26.223592347 +0000 UTC m=+4891.908140281" watchObservedRunningTime="2026-06-02 17:08:26.224139677 +0000 UTC m=+4891.908687621"
Jun 02 17:08:26.619827 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:26.619731    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xfr2t_c59b270b-1208-4a9a-bd68-2ad2507ebf2f/global-pull-secret-syncer/0.log"
Jun 02 17:08:26.744182 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:26.744150    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pwm24_19b470d3-1650-4994-9a8d-351b7ce6f3c5/konnectivity-agent/0.log"
Jun 02 17:08:26.766071 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:26.766017    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-190.ec2.internal_c0408568ef573e264a6bee7a242559ac/haproxy/0.log"
Jun 02 17:08:28.566804 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.566764    2568 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bj4rc/must-gather-jdd4w"]
Jun 02 17:08:28.569991 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.569957    2568 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bj4rc/must-gather-jdd4w"]
Jun 02 17:08:28.570155 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.570083    2568 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="copy" containerID="cri-o://4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a" gracePeriod=2
Jun 02 17:08:28.575069 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.572155    2568 status_manager.go:895] "Failed to get status for pod" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" pod="openshift-must-gather-bj4rc/must-gather-jdd4w" err="pods \"must-gather-jdd4w\" is forbidden: User \"system:node:ip-10-0-135-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bj4rc\": no relationship found between node 'ip-10-0-135-190.ec2.internal' and this object"
Jun 02 17:08:28.964359 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.964268    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bj4rc_must-gather-jdd4w_4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778/copy/0.log"
Jun 02 17:08:28.966392 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:28.966257    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:08:29.020979 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.020914    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh2wg\" (UniqueName: \"kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg\") pod \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") "
Jun 02 17:08:29.021282 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.021246    2568 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output\") pod \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\" (UID: \"4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778\") "
Jun 02 17:08:29.026495 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.026453    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg" (OuterVolumeSpecName: "kube-api-access-kh2wg") pod "4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" (UID: "4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778"). InnerVolumeSpecName "kube-api-access-kh2wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:08:29.044068 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.041446    2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" (UID: "4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:29.122446 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.122368    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2wg\" (UniqueName: \"kubernetes.io/projected/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-kube-api-access-kh2wg\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:08:29.122446 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.122408    2568 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778-must-gather-output\") on node \"ip-10-0-135-190.ec2.internal\" DevicePath \"\""
Jun 02 17:08:29.224348 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.224274    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bj4rc_must-gather-jdd4w_4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778/copy/0.log"
Jun 02 17:08:29.224951 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.224926    2568 generic.go:358] "Generic (PLEG): container finished" podID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerID="4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a" exitCode=143
Jun 02 17:08:29.225813 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.225197    2568 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bj4rc/must-gather-jdd4w"
Jun 02 17:08:29.226347 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.225229    2568 scope.go:117] "RemoveContainer" containerID="4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a"
Jun 02 17:08:29.236320 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.236253    2568 scope.go:117] "RemoveContainer" containerID="d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"
Jun 02 17:08:29.265760 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.265736    2568 scope.go:117] "RemoveContainer" containerID="4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a"
Jun 02 17:08:29.268597 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:08:29.266126    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a\": container with ID starting with 4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a not found: ID does not exist" containerID="4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a"
Jun 02 17:08:29.268597 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.266165    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a"} err="failed to get container status \"4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a\": rpc error: code = NotFound desc = could not find container \"4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a\": container with ID starting with 4ebca7d22018cff6039e346a22abc7b8184b111f627c0ae7c4ed4fdf33ac553a not found: ID does not exist"
Jun 02 17:08:29.268597 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.266190    2568 scope.go:117] "RemoveContainer" containerID="d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"
Jun 02 17:08:29.268597 ip-10-0-135-190 kubenswrapper[2568]: E0602 17:08:29.266479    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3\": container with ID starting with d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3 not found: ID does not exist" containerID="d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"
Jun 02 17:08:29.268597 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:29.266513    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3"} err="failed to get container status \"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3\": rpc error: code = NotFound desc = could not find container \"d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3\": container with ID starting with d298d03cb31ffe537d3118ce1f952359ea61ac3117ad91a9e15c9f57e5c0e0a3 not found: ID does not exist"
Jun 02 17:08:30.895699 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:30.895330    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" path="/var/lib/kubelet/pods/4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778/volumes"
Jun 02 17:08:30.968551 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:30.968524    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-2xtlj_6bc67b1a-a154-4db0-8694-70287f3aceac/manager/0.log"
Jun 02 17:08:31.000180 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:31.000145    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-v6vt8_8f5aa1ff-a62d-428b-b5b5-8bd5b7e40f7f/kuadrant-console-plugin/0.log"
Jun 02 17:08:32.223354 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.223293    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-6zlfb_69f3fc2c-9f29-43df-9567-93e2e8eaec22/kube-state-metrics/0.log"
Jun 02 17:08:32.245225 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.245164    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-6zlfb_69f3fc2c-9f29-43df-9567-93e2e8eaec22/kube-rbac-proxy-main/0.log"
Jun 02 17:08:32.268215 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.268165    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-6zlfb_69f3fc2c-9f29-43df-9567-93e2e8eaec22/kube-rbac-proxy-self/0.log"
Jun 02 17:08:32.300672 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.300587    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8645bc875b-nbzvh_10097207-4680-4999-9bcd-776f85ae34af/metrics-server/0.log"
Jun 02 17:08:32.380558 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.380526    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vswl_8a67760f-b952-4ee6-b439-b13ce2461ca6/node-exporter/0.log"
Jun 02 17:08:32.413617 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.403675    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vswl_8a67760f-b952-4ee6-b439-b13ce2461ca6/kube-rbac-proxy/0.log"
Jun 02 17:08:32.437961 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:32.437933    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vswl_8a67760f-b952-4ee6-b439-b13ce2461ca6/init-textfile/0.log"
Jun 02 17:08:35.324539 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.324512    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6ccc75765d-xjp79_fb2b9c87-cf68-4d56-9c70-815b07e87998/console/0.log"
Jun 02 17:08:35.439645 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.439602    2568 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"]
Jun 02 17:08:35.440129 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440102    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="copy"
Jun 02 17:08:35.440129 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440128    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="copy"
Jun 02 17:08:35.440310 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440145    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="gather"
Jun 02 17:08:35.440310 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440155    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="gather"
Jun 02 17:08:35.440310 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440240    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="gather"
Jun 02 17:08:35.440310 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.440256    2568 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b8ed89d-4f7f-4d63-ac0d-8c24cf18e778" containerName="copy"
Jun 02 17:08:35.444458 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.444429    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.452596 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.452564    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"]
Jun 02 17:08:35.588594 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.588508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-sys\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.588594 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.588557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjkk\" (UniqueName: \"kubernetes.io/projected/1a12743c-f956-4025-b55b-22fb5e8c57f4-kube-api-access-rhjkk\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.588972 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.588682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-lib-modules\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.588972 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.588741    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-proc\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.588972 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.588777    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-podres\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689572 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689532    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-sys\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689572 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689575    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjkk\" (UniqueName: \"kubernetes.io/projected/1a12743c-f956-4025-b55b-22fb5e8c57f4-kube-api-access-rhjkk\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689639    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-lib-modules\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689661    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-sys\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689683    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-proc\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.689805 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689733    2568 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-podres\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.690115 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689834    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-proc\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.690115 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689854    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-lib-modules\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.690115 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.689881    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12743c-f956-4025-b55b-22fb5e8c57f4-podres\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.697812 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.697784    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjkk\" (UniqueName: \"kubernetes.io/projected/1a12743c-f956-4025-b55b-22fb5e8c57f4-kube-api-access-rhjkk\") pod \"perf-node-gather-daemonset-kmks4\" (UID: \"1a12743c-f956-4025-b55b-22fb5e8c57f4\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.759402 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.758958    2568 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:35.913656 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:35.912900    2568 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"]
Jun 02 17:08:35.916674 ip-10-0-135-190 kubenswrapper[2568]: W0602 17:08:35.916159    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a12743c_f956_4025_b55b_22fb5e8c57f4.slice/crio-94f0a705192fb6fcdc288b6acc218d24c0482a1fdafbef1b1c4756cf4966c640 WatchSource:0}: Error finding container 94f0a705192fb6fcdc288b6acc218d24c0482a1fdafbef1b1c4756cf4966c640: Status 404 returned error can't find the container with id 94f0a705192fb6fcdc288b6acc218d24c0482a1fdafbef1b1c4756cf4966c640
Jun 02 17:08:36.263294 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.263255    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4" event={"ID":"1a12743c-f956-4025-b55b-22fb5e8c57f4","Type":"ContainerStarted","Data":"ed34e8fc7fe1ae0d15caeca24d75fd73703ce2137be43fddb7e24611b90c7a95"}
Jun 02 17:08:36.263294 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.263300    2568 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4" event={"ID":"1a12743c-f956-4025-b55b-22fb5e8c57f4","Type":"ContainerStarted","Data":"94f0a705192fb6fcdc288b6acc218d24c0482a1fdafbef1b1c4756cf4966c640"}
Jun 02 17:08:36.263531 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.263486    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:36.278447 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.278238    2568 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4" podStartSLOduration=1.278221672 podStartE2EDuration="1.278221672s" podCreationTimestamp="2026-06-02 17:08:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:08:36.277689915 +0000 UTC m=+4901.962237851" watchObservedRunningTime="2026-06-02 17:08:36.278221672 +0000 UTC m=+4901.962769608"
Jun 02 17:08:36.558135 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.558066    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zqcd_dc37083d-acbb-4bf3-b507-84bf54dd939d/dns/0.log"
Jun 02 17:08:36.579228 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.579198    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zqcd_dc37083d-acbb-4bf3-b507-84bf54dd939d/kube-rbac-proxy/0.log"
Jun 02 17:08:36.688176 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:36.688152    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gltw9_099e71f1-a837-487b-a2e1-70e9e992800a/dns-node-resolver/0.log"
Jun 02 17:08:37.190653 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:37.190618    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tqbr4_72ba1511-d074-4216-ad97-1459127915d0/node-ca/0.log"
Jun 02 17:08:38.552099 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:38.552075    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b98jg_e9c5a910-06ea-4bd4-9afb-a5c61d74f572/serve-healthcheck-canary/0.log"
Jun 02 17:08:39.171702 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:39.171671    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9cxq_4968d9d2-480a-406d-85c6-ae2468b7e837/kube-rbac-proxy/0.log"
Jun 02 17:08:39.191610 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:39.191582    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9cxq_4968d9d2-480a-406d-85c6-ae2468b7e837/exporter/0.log"
Jun 02 17:08:39.210629 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:39.210593    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z9cxq_4968d9d2-480a-406d-85c6-ae2468b7e837/extractor/0.log"
Jun 02 17:08:41.693462 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:41.693429    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74c68556d6-9lvm9_dfb2eeab-6020-44f9-bd7e-f4c64689aecd/manager/0.log"
Jun 02 17:08:41.738025 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:41.737988    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-m947k_24f3516f-8dab-48fe-8649-940e3d482501/openshift-lws-operator/0.log"
Jun 02 17:08:42.279279 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:42.279245    2568 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-kmks4"
Jun 02 17:08:42.322249 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:42.322218    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-86c7bf9457-dxv9d_94b238f0-5cab-4f2f-96d3-df407f18629c/manager/0.log"
Jun 02 17:08:42.339917 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:42.339888    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-8wvqq_e357b82e-e589-4444-b0e3-46f45ae21be1/server/0.log"
Jun 02 17:08:42.539078 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:42.538982    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7rjr7_3c9e393d-fadc-4ae8-93ac-254bdef49132/s3-init/0.log"
Jun 02 17:08:48.743758 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.743732    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/kube-multus-additional-cni-plugins/0.log"
Jun 02 17:08:48.763502 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.763475    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/egress-router-binary-copy/0.log"
Jun 02 17:08:48.785790 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.785728    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/cni-plugins/0.log"
Jun 02 17:08:48.805533 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.805512    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/bond-cni-plugin/0.log"
Jun 02 17:08:48.824295 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.824276    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/routeoverride-cni/0.log"
Jun 02 17:08:48.845020 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.845001    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/whereabouts-cni-bincopy/0.log"
Jun 02 17:08:48.863357 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.863340    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knr5w_e07b584a-2b02-442f-80f5-e5d7d89f1452/whereabouts-cni/0.log"
Jun 02 17:08:48.959495 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:48.959466    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x2vql_10fc7050-27b5-48fa-9b5b-5260a5bedf0a/kube-multus/0.log"
Jun 02 17:08:49.067271 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:49.067187    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xtvfm_c0186ca7-bafe-4ba4-82bf-0257fbff4133/network-metrics-daemon/0.log"
Jun 02 17:08:49.085860 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:49.085828    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xtvfm_c0186ca7-bafe-4ba4-82bf-0257fbff4133/kube-rbac-proxy/0.log"
Jun 02 17:08:50.609541 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.609515    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-controller/0.log"
Jun 02 17:08:50.625488 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.625436    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/0.log"
Jun 02 17:08:50.649190 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.649168    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovn-acl-logging/1.log"
Jun 02 17:08:50.667475 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.667448    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/kube-rbac-proxy-node/0.log"
Jun 02 17:08:50.691381 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.691361    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 17:08:50.707495 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.707470    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/northd/0.log"
Jun 02 17:08:50.727554 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.727536    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/nbdb/0.log"
Jun 02 17:08:50.746948 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.746929    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/sbdb/0.log"
Jun 02 17:08:50.889717 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:50.889683    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n79m7_0e6bad57-9fda-42cd-9570-a90fec836696/ovnkube-controller/0.log"
Jun 02 17:08:51.956266 ip-10-0-135-190 kubenswrapper[2568]: I0602 17:08:51.956234    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-69vh8_2c88dfd2-651a-4626-a690-35e05cc1ba9d/network-check-target-container/0.log"