May 26 11:25:03.907754 ip-10-0-133-125 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:25:04.409873 ip-10-0-133-125 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:25:04.409873 ip-10-0-133-125 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:25:04.409873 ip-10-0-133-125 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:25:04.409873 ip-10-0-133-125 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:25:04.409873 ip-10-0-133-125 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:25:04.411911 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.411765    2562 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:25:04.415962 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415946    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:25:04.415962 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415962    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415967    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415973    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415978    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415981    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415991    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415995    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.415997    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416001    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416004    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416006    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416009    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416012    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416015    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416018    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416021    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416023    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416026    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416029    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:25:04.416026 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416032    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416035    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416038    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416041    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416044    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416046    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416048    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416051    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416053    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416056    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416058    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416061    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416063    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416066    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416068    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416070    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416073    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416075    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416078    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:25:04.416609 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416081    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416084    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416086    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416089    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416091    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416094    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416096    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416099    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416101    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416103    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416106    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416108    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416111    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416113    2562 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416116    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416121    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416124    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416128    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416131    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416135    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:25:04.417070 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416138    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416141    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416144    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416147    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416149    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416152    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416155    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416158    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416160    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416163    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416165    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416168    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416171    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416174    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416176    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416179    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416181    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416184    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416186    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416189    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:25:04.417557 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416191    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416194    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416197    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416199    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416202    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416205    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:25:04.418028 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.416208    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:25:04.418667 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418655    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:25:04.418667 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418667    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418670    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418674    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418678    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418682    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418685    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418688    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418691    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418694    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418697    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418699    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418702    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418704    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418707    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418710    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418712    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418715    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418718    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418721    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:25:04.418729 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418724    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418726    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418729    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418732    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418735    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418738    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418740    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418743    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418746    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418748    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418751    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418753    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418757    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418759    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418761    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418764    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418766    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418769    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418771    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:25:04.419182 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418774    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418776    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418779    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418781    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418786    2562 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418788    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418790    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418793    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418795    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418798    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418800    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418803    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418807    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418810    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418812    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418816    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418818    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418821    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418823    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418826    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:25:04.419696 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418828    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418831    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418833    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418836    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418838    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418841    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418845    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418849    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418852    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418854    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418857    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418859    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418862    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418864    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418866    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418869    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418872    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418874    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418877    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:25:04.420184 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418880    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418882    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418885    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418887    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418890    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418893    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418896    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.418898    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.418971    2562 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.418978    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.418985    2562 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.418990    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419007    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419011    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419016    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419020    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419024    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419028    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419032    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419035    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419039    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419042    2562 flags.go:64] FLAG: --cgroup-root=""
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419045    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:25:04.420652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419048    2562 flags.go:64] FLAG: --client-ca-file=""
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419051    2562 flags.go:64] FLAG: --cloud-config=""
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419054    2562 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419057    2562 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419061    2562 flags.go:64] FLAG: --cluster-domain=""
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419064    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419067    2562 flags.go:64] FLAG: --config-dir=""
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419070    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419074    2562 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419082    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419085    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419088    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419091    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419094    2562 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419097    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419100    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419109    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419112    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419117    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419120    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419123    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419126    2562 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419129    2562 flags.go:64] FLAG: --enable-server="true"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419131    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419137    2562 flags.go:64] FLAG: --event-burst="100"
May 26 11:25:04.421218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419140    2562 flags.go:64] FLAG: --event-qps="50"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419143    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419146    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419149    2562 flags.go:64] FLAG: --eviction-hard=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419152    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419155    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419158    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419161    2562 flags.go:64] FLAG: --eviction-soft=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419165    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419168    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419171    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419174    2562 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419177    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419180    2562 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419182    2562 flags.go:64] FLAG: --feature-gates=""
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419187    2562 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419191    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419195    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419198    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419201    2562 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419204    2562 flags.go:64] FLAG: --help="false"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419207    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419210    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419213    2562 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:25:04.421829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419216    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419219    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419223    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419226    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419228    2562 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419231    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419234    2562 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419237    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419240    2562 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419245    2562 flags.go:64] FLAG: --kube-reserved=""
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419248    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419250    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419254    2562 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419256    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419259    2562 flags.go:64] FLAG: --lock-file=""
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419262    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419265    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419268    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419273    2562 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419276    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419279    2562 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419281    2562 flags.go:64] FLAG: --logging-format="text"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419284    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419287    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:25:04.422406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419290    2562 flags.go:64] FLAG: --manifest-url=""
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419293    2562 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419297    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419300    2562 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419304    2562 flags.go:64] FLAG: --max-pods="110"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419307    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419310    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419313    2562 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419316    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419320    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419323    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419326    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419333    2562 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419336    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419339    2562 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419343    2562 flags.go:64] FLAG: --pod-cidr=""
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419345    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419352    2562 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419355    2562 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419359    2562 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419362    2562 flags.go:64] FLAG: --port="10250"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419365    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419368    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fdcef2f2faecffb2"
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419371    2562 flags.go:64] FLAG: --qos-reserved=""
May 26 11:25:04.422998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419374    2562 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419377    2562 flags.go:64] FLAG: --register-node="true"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419380    2562 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419383    2562 flags.go:64] FLAG: --register-with-taints=""
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419387    2562 flags.go:64] FLAG: --registry-burst="10"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419390    2562 flags.go:64] FLAG: --registry-qps="5"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419393    2562 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419395    2562 flags.go:64] FLAG: --reserved-memory=""
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419399    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419402    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419405    2562 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419408    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419425    2562 flags.go:64] FLAG: --runonce="false"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419429    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419432    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419435    2562 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419438    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419441    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419445    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419448    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419451    2562 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419454    2562 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419457    2562 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419460    2562 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419463    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419468    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:25:04.423580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419471    2562 flags.go:64] FLAG: --system-cgroups=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419473    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419479    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419481    2562 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419484    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419488    2562 flags.go:64] FLAG: --tls-min-version=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419491    2562 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419494    2562 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419497    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419500    2562 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419503    2562 flags.go:64] FLAG: --v="2"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419507    2562 flags.go:64] FLAG: --version="false"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419511    2562 flags.go:64] FLAG: --vmodule=""
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419516    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419519    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419613    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419616    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419619    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419622    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419625    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419627    2562 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419630    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419632    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:25:04.424189 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419635    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419637    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419640    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419643    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419646    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419648    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419651    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419653    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419657    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419659    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419662    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419664    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419667    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419670    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419672    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419674    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419677    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419679    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419681    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419684    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:25:04.424790 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419686    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419689    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419691    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419695    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419698    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419700    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419703    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419705    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419708    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419711    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419713    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419716    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419720    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419723    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419726    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419729    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419731    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419734    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419736    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:25:04.425322 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419739    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419742    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419744    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419747    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419749    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419752    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419755    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419757    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419759    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419762    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419764    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419767    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419769    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419772    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419774    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419777    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419781    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419784    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419786    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419789    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:25:04.425916 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419792    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419794    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419796    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419799    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419802    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419804    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419807    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419810    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419812    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419815    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419819    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419822    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419825    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419828    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419831    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419833    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419836    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419838    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:25:04.426437 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.419841    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:25:04.427018 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.419847    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:25:04.429599 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.429569    2562 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:25:04.429599 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.429600    2562 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429652    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429659    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429662    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429665    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429668    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429671    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429673    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429676    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429679    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429682    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429686    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429688    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429691    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429694    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429697    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429699    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429702    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429704    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429707    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:25:04.429726 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429709    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429712    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429715    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429718    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429720    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429724    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429727    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429730    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429733    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429736    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429738    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429741    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429744    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429747    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429749    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429752    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429754    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429757    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429759    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429762    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:25:04.430224 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429764    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429767    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429769    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429772    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429774    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429777    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429780    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429782    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429785    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429788    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429791    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429793    2562 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429796    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429799    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429802    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429804    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429807    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429809    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429812    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429814    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:25:04.430734 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429817    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429820    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429822    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429825    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429829    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429833    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429836    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429839    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429842    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429845    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429849    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429853    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429856    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429859    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429862    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429865    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429867    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429871    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:25:04.431245 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429873    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429876    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429878    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429881    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429884    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429887    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429889    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429893    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429895    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.429901    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.429997    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430002    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430006    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430009    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430011    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:25:04.431709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430014    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430017    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430020    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430023    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430026    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430028    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430031    2562 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430034    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430036    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430039    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430041    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430044    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430046    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430050    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430054    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430057    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430059    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430062    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430065    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430068    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:25:04.432076 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430070    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430072    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430076    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430078    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430081    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430084    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430087    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430089    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430092    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430094    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430097    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430099    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430102    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430105    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430107    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430110    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430112    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430115    2562 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430117    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:25:04.432646 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430120    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430122    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430125    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430127    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430129    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430132    2562 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430134    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430137    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430139    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430142    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430144    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430147    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430150    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430152    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430155    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430157    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430160    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430163    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430165    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430168    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:25:04.433097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430170    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430173    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430176    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430178    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430182    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430185    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430189    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430192    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430194    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430198    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430200    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430203    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430206    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430208    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430211    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430213    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430216    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430218    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430221    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:25:04.433583 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430224    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:25:04.434045 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430226    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:25:04.434045 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:04.430229    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:25:04.434045 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.430233    2562 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:25:04.434045 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.431113    2562 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:25:04.436354 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.436339    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:25:04.437504 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.437492    2562 server.go:1019] "Starting client certificate rotation"
May 26 11:25:04.437618 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.437600    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:25:04.437677 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.437650    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:25:04.465884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.465857    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:25:04.467819 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.467796    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:25:04.481884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.481868    2562 log.go:25] "Validated CRI v1 runtime API"
May 26 11:25:04.490501 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.490484    2562 log.go:25] "Validated CRI v1 image API"
May 26 11:25:04.492641 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.492623    2562 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:25:04.497128 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.497106    2562 fs.go:135] Filesystem UUIDs: map[1f4b0b07-578f-457b-959e-2af2b85324ff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bd362a14-c0bf-4b55-9beb-3cae7263de0a:/dev/nvme0n1p3]
May 26 11:25:04.497212 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.497127    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:25:04.499670 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.499653    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:25:04.501328 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.501219    2562 manager.go:217] Machine: {Timestamp:2026-05-26 11:25:04.500737858 +0000 UTC m=+0.461285333 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100344 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29d4571dd63252755444ed01865995 SystemUUID:ec29d457-1dd6-3252-7554-44ed01865995 BootID:c2ca1392-d4f2-4121-9938-fc588d3cdac6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:f9:0f:09:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:f9:0f:09:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:52:37:cb:fe:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:25:04.501993 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.501981    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:25:04.502080 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.502067    2562 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:25:04.505008 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.504979    2562 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:25:04.505144 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.505012    2562 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:25:04.505185 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.505150    2562 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:25:04.505185 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.505160    2562 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:25:04.505185 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.505166    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:25:04.505185 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.505175    2562 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:25:04.507206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.507195    2562 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:25:04.507317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.507308    2562 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:25:04.510073 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.510063    2562 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:25:04.510107 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.510077    2562 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:25:04.510107 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.510090    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:25:04.510107 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.510098    2562 kubelet.go:397] "Adding apiserver pod source"
May 26 11:25:04.510107 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.510107    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:25:04.511281 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.511269    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:25:04.511343 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.511288    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:25:04.516195 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.516170    2562 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:25:04.518616 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.518601    2562 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:25:04.519647 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519634    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519655    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519664    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519672    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519684    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519692    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:25:04.519702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519703    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:25:04.519865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519712    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:25:04.519865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519722    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:25:04.519865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519731    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:25:04.519865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.519744    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:25:04.520177 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.520163    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:25:04.521189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.521179    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:25:04.521189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.521189    2562 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:25:04.522225 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.522206    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:25:04.522225 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.522208    2562 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:25:04.524450 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524437    2562 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:25:04.524501 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524472    2562 server.go:1295] "Started kubelet"
May 26 11:25:04.524568 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524538    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:25:04.524624 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524565    2562 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:25:04.524672 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524631    2562 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:25:04.525014 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.524995    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j6wg"
May 26 11:25:04.525333 ip-10-0-133-125 systemd[1]: Started Kubernetes Kubelet.
May 26 11:25:04.525826 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.525718    2562 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:25:04.531343 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.531318    2562 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:25:04.532714 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.532690    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j6wg"
May 26 11:25:04.534027 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.534010    2562 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:25:04.536537 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.534231    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-125.ec2.internal.18b319daa04d1354  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-125.ec2.internal,UID:ip-10-0-133-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-125.ec2.internal,},FirstTimestamp:2026-05-26 11:25:04.524448596 +0000 UTC m=+0.484996070,LastTimestamp:2026-05-26 11:25:04.524448596 +0000 UTC m=+0.484996070,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-125.ec2.internal,}"
May 26 11:25:04.537695 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.537672    2562 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:25:04.539827 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.539810    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:25:04.540511 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540486    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:25:04.540637 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540620    2562 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:25:04.540637 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540637    2562 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:25:04.540775 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540743    2562 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:25:04.540842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540825    2562 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:25:04.540842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.540838    2562 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:25:04.541134 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541117    2562 factory.go:55] Registering systemd factory
May 26 11:25:04.541134 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541136    2562 factory.go:223] Registration of the systemd container factory successfully
May 26 11:25:04.541330 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.541301    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:04.541442 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541366    2562 factory.go:153] Registering CRI-O factory
May 26 11:25:04.541442 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541394    2562 factory.go:223] Registration of the crio container factory successfully
May 26 11:25:04.541545 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541466    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:25:04.541545 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541488    2562 factory.go:103] Registering Raw factory
May 26 11:25:04.541545 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.541507    2562 manager.go:1196] Started watching for new ooms in manager
May 26 11:25:04.542949 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.542931    2562 manager.go:319] Starting recovery of all containers
May 26 11:25:04.545250 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.545225    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:04.548656 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.548638    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.552250 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.552117    2562 manager.go:324] Recovery completed
May 26 11:25:04.556683 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.556671    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.558343 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558327    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.558440 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558360    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.558440 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558383    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.558910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558896    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:25:04.558910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558908    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:25:04.558998 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.558927    2562 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:25:04.562774 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.562762    2562 policy_none.go:49] "None policy: Start"
May 26 11:25:04.562828 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.562779    2562 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:25:04.562828 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.562790    2562 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:25:04.614111 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614093    2562 manager.go:341] "Starting Device Plugin manager"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.614128    2562 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614139    2562 server.go:85] "Starting device plugin registration server"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614437    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614448    2562 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614588    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614699    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.614710    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.615790    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:25:04.625814 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.615829    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:04.672565 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.672481    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:25:04.673621 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.673601    2562 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:25:04.673699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.673639    2562 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:25:04.673699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.673665    2562 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:25:04.673699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.673676    2562 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:25:04.673842 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.673720    2562 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:25:04.680377 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.680355    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:04.715620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.715592    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.716606 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.716588    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.716719 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.716623    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.716719 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.716637    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.716719 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.716667    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.725025 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.725006    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.725111 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.725030    2562 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-125.ec2.internal\": node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:04.753785 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.753762    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:04.774345 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.774301    2562 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"]
May 26 11:25:04.774508 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.774402    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.775857 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.775831    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.775944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.775863    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.775944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.775873    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.776827 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.776815    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.776946 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.776930    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.776982 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.776960    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.777469 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777455    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.777548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777478    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.777548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777488    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.777548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777494    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.777548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777511    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.777548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.777521    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.778484 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.778470    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.778561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.778496    2562 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:25:04.779117 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.779104    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:25:04.779203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.779131    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:25:04.779203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.779144    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
May 26 11:25:04.803100 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.803079    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.807506 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.807491    2562 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal"
May 26 11:25:04.842248 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.842220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.842362 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.842254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.842362 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.842279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f88796f630f0b1df3d5abb08c2fec007-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"f88796f630f0b1df3d5abb08c2fec007\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.853982 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.853960    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:04.943359 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943299    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f88796f630f0b1df3d5abb08c2fec007-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"f88796f630f0b1df3d5abb08c2fec007\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.943359 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943334    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f88796f630f0b1df3d5abb08c2fec007-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"f88796f630f0b1df3d5abb08c2fec007\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.943474 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943381    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.943474 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943401    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.943474 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943444    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.943474 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:04.943459    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839c040c48a39333914f45b99d70dd25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"839c040c48a39333914f45b99d70dd25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:04.954069 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:04.954038    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.054439 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.054381    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.105582 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.105557    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:05.110252 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.110235    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:05.154790 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.154759    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.255295 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.255217    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.355753 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.355719    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.437944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.437917    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:25:05.438386 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.438076    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:25:05.438386 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.438092    2562 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:25:05.456400 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.456373    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.536756 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.536681    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:20:04 +0000 UTC" deadline="2027-11-02 14:33:47.291102965 +0000 UTC"
May 26 11:25:05.536756 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.536710    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12603h8m41.754396586s"
May 26 11:25:05.540570 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.540548    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:25:05.552692 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.552668    2562 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:25:05.557152 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:05.557130    2562 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
May 26 11:25:05.575166 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.575146    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxzvh"
May 26 11:25:05.583375 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.583350    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxzvh"
May 26 11:25:05.606106 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:05.606074    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839c040c48a39333914f45b99d70dd25.slice/crio-6e0196a94ed7c1cfdc3fcfe1df72afff7684bff6eaa1b73e8b20eb2602478113 WatchSource:0}: Error finding container 6e0196a94ed7c1cfdc3fcfe1df72afff7684bff6eaa1b73e8b20eb2602478113: Status 404 returned error can't find the container with id 6e0196a94ed7c1cfdc3fcfe1df72afff7684bff6eaa1b73e8b20eb2602478113
May 26 11:25:05.606534 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:05.606496    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88796f630f0b1df3d5abb08c2fec007.slice/crio-16adb8fca6b388582c215b796a223bc824037a6513f783771128821e9196c05a WatchSource:0}: Error finding container 16adb8fca6b388582c215b796a223bc824037a6513f783771128821e9196c05a: Status 404 returned error can't find the container with id 16adb8fca6b388582c215b796a223bc824037a6513f783771128821e9196c05a
May 26 11:25:05.611005 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.610992    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:25:05.630746 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.630724    2562 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:05.641581 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.641557    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
May 26 11:25:05.652193 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.652173    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:25:05.654294 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.654281    2562 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
May 26 11:25:05.668946 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.668922    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:25:05.676956 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.676915    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"839c040c48a39333914f45b99d70dd25","Type":"ContainerStarted","Data":"6e0196a94ed7c1cfdc3fcfe1df72afff7684bff6eaa1b73e8b20eb2602478113"}
May 26 11:25:05.677876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.677857    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"f88796f630f0b1df3d5abb08c2fec007","Type":"ContainerStarted","Data":"16adb8fca6b388582c215b796a223bc824037a6513f783771128821e9196c05a"}
May 26 11:25:05.792684 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:05.792601    2562 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:06.511347 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.511314    2562 apiserver.go:52] "Watching apiserver"
May 26 11:25:06.517988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.517964    2562 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:25:06.520289 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.520256    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-z2bzc","openshift-multus/multus-cjv66","openshift-multus/network-metrics-daemon-hkf6m","openshift-cluster-node-tuning-operator/tuned-pjhw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal","openshift-network-diagnostics/network-check-target-6866p","openshift-network-operator/iptables-alerter-h4pkj","openshift-ovn-kubernetes/ovnkube-node-b8trm","kube-system/konnectivity-agent-bxkxv","kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl","openshift-dns/node-resolver-lsn6g","openshift-image-registry/node-ca-f8m2x"]
May 26 11:25:06.522809 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.522783    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.525355 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.525332    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78lvm\""
May 26 11:25:06.525483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.525341    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:25:06.525851 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.525825    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:25:06.525954 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.525408    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:25:06.526337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.526319    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjv66"
May 26 11:25:06.528391 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.528341    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.528504 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.528440    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:06.529026 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.529001    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:25:06.529143 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.529120    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:25:06.529241 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.529213    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:25:06.529302 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.529269    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vpvdv\""
May 26 11:25:06.529358 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.529311    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:25:06.531181 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.531160    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.533030 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.532986    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:25:06.533261 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.533229    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:25:06.533351 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.533338    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpn7p\""
May 26 11:25:06.535398 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.535349    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:06.535555 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.535453    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.535555 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.535444    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:06.537336 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.537316    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:25:06.537519 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.537500    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sttsk\""
May 26 11:25:06.537766 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.537750    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.537997 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.537750    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:25:06.539721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.539667    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:25:06.541009 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.540863    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:25:06.541009 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.540886    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:25:06.541009 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.540886    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:25:06.541009 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.540965    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:25:06.541235 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.541171    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:25:06.541365 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.541345    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bk85b\""
May 26 11:25:06.542334 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.542318    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.542508 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.542491    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.544252 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.544213    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:25:06.544571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.544531    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:25:06.544775 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.544758    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.545068 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.544975    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:25:06.545068 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.544974    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mfb4g\""
May 26 11:25:06.545217 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.545081    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:25:06.545217 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.545133    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r65rh\""
May 26 11:25:06.545331 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.545255    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:25:06.546696 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.546676    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vlcw4\""
May 26 11:25:06.546976 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.546959    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:25:06.547076 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.547059    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.547221 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.547204    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:25:06.548979 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.548962    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:25:06.549165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.549145    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7gwwf\""
May 26 11:25:06.549266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.549201    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:25:06.549266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.549255    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:25:06.551865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.551837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-tmp\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.552001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.551872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:06.552001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.551899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.552001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.551961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-conf-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.552001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.551995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-kubernetes\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-bin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-hostroot\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-system-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-socket-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-modprobe-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.552206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfn9m\" (UniqueName: \"kubernetes.io/projected/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-kube-api-access-hfn9m\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.552508 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-log-socket\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.552508 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.552508 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-etc-selinux\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.552639 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-daemon-config\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.552685 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysconfig\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.552734 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-run\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.552734 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-etc-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.552823 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-env-overrides\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.552823 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwv2t\" (UniqueName: \"kubernetes.io/projected/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-kube-api-access-fwv2t\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.552912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-host-slash\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.552912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cnibin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553002 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-etc-kubernetes\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553002 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.552972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-lib-modules\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.553096 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-systemd-units\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.553096 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-node-log\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.553190 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-k8s-cni-cncf-io\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553190 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-systemd\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.553190 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.553327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9d5r\" (UniqueName: \"kubernetes.io/projected/b405a8c5-b718-4ec6-bebe-2f1209fe5859-kube-api-access-w9d5r\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.553327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-systemd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.553327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-ovn\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.553327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cni-binary-copy\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-multus\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-kubelet\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-registration-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-iptables-alerter-script\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rszt\" (UniqueName: \"kubernetes.io/projected/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-kube-api-access-6rszt\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-multus-certs\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-socket-dir-parent\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.553594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-conf\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.553938 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mgw\" (UniqueName: \"kubernetes.io/projected/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-kube-api-access-p6mgw\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.553938 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.553712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-os-release\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-device-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-sys\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-kubelet\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-netns\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-netd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-script-lib\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-sys-fs\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-tuned\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cnibin\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-slash\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-bin\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-config\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-os-release\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.555910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-system-cni-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-var-lib-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovn-node-metrics-cert\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.554975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvl6w\" (UniqueName: \"kubernetes.io/projected/961fc510-7c65-49b2-a4b3-b193252a6268-kube-api-access-vvl6w\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbm4\" (UniqueName: \"kubernetes.io/projected/50f2f01d-8199-4e84-ab1b-96c4a1a06803-kube-api-access-mkbm4\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-var-lib-kubelet\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c466082b-97a7-404a-91c0-8c9ac1e9c791-agent-certs\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c466082b-97a7-404a-91c0-8c9ac1e9c791-konnectivity-ca\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-netns\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.556590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.555290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-host\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.584041 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.584006    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:20:05 +0000 UTC" deadline="2028-03-02 15:25:36.432235024 +0000 UTC"
May 26 11:25:06.584041 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.584035    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15508h0m29.848203176s"
May 26 11:25:06.642098 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.642071    2562 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:25:06.655960 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.655930    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvl6w\" (UniqueName: \"kubernetes.io/projected/961fc510-7c65-49b2-a4b3-b193252a6268-kube-api-access-vvl6w\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.656122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.655966    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbm4\" (UniqueName: \"kubernetes.io/projected/50f2f01d-8199-4e84-ab1b-96c4a1a06803-kube-api-access-mkbm4\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.655997    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-var-lib-kubelet\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656021    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.656122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656072    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c466082b-97a7-404a-91c0-8c9ac1e9c791-agent-certs\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.656122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656104    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-var-lib-kubelet\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656108    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c466082b-97a7-404a-91c0-8c9ac1e9c791-konnectivity-ca\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.656366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656160    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656179    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-netns\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656202    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-host\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656248    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-tmp\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656367    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656367    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-netns\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656429    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-host\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656444    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656459    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656478    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656503    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-conf-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656527    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-kubernetes\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656552    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-bin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.656576    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656577    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-hostroot\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656582    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-conf-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.656620 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656626    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c466082b-97a7-404a-91c0-8c9ac1e9c791-konnectivity-ca\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb390ce-0700-4745-9dda-fe590c8284c2-host\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656664    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-kubernetes\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656669    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656627    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-hostroot\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656689    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-bin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.656741    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:07.156629763 +0000 UTC m=+3.117177250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656792    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-system-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656812    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-socket-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656862    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-modprobe-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656875    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-system-cni-dir\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656901    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656929    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hfn9m\" (UniqueName: \"kubernetes.io/projected/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-kube-api-access-hfn9m\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656952    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-log-socket\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656977    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657003    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-etc-selinux\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.657207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657010    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-socket-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657021    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-log-socket\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657030    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-daemon-config\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.656981    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-modprobe-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657041    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657054    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysconfig\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657076    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-run\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657100    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-etc-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657121    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysconfig\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657124    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-env-overrides\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657174    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fwv2t\" (UniqueName: \"kubernetes.io/projected/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-kube-api-access-fwv2t\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657114    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-etc-selinux\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657176    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-run\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657202    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-host-slash\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657203    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-etc-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657075    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657252    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-host-slash\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657321    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cnibin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.657940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657347    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-etc-kubernetes\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657378    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-lib-modules\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657403    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cnibin\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657447    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-etc-kubernetes\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6197213-20ec-4de9-8308-3f48b0cee933-tmp-dir\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657501    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-lib-modules\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657516    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-systemd-units\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657552    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-systemd-units\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657554    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-daemon-config\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657589    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-node-log\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657599    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-env-overrides\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657616    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-k8s-cni-cncf-io\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657637    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-node-log\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657642    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-systemd\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657662    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-k8s-cni-cncf-io\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdxz\" (UniqueName: \"kubernetes.io/projected/c6197213-20ec-4de9-8308-3f48b0cee933-kube-api-access-ksdxz\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657694    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-systemd\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657703    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.658772 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657723    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w9d5r\" (UniqueName: \"kubernetes.io/projected/b405a8c5-b718-4ec6-bebe-2f1209fe5859-kube-api-access-w9d5r\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657745    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-systemd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657776    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-systemd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657852    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-ovn\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657876    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cni-binary-copy\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657935    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-ovn\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657956    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-multus\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.657985    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-kubelet\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cb390ce-0700-4745-9dda-fe590c8284c2-serviceca\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658032    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-cni-multus\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658037    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-registration-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658079    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-var-lib-kubelet\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658086    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-iptables-alerter-script\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658123    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6rszt\" (UniqueName: \"kubernetes.io/projected/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-kube-api-access-6rszt\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658126    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-registration-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658152    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-multus-certs\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658180    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.659629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658206    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-socket-dir-parent\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658228    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-conf\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658253    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mgw\" (UniqueName: \"kubernetes.io/projected/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-kube-api-access-p6mgw\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658278    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-os-release\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658291    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658303    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658329    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-device-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658354    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-sys\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wf4\" (UniqueName: \"kubernetes.io/projected/5cb390ce-0700-4745-9dda-fe590c8284c2-kube-api-access-q6wf4\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658392    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-multus-socket-dir-parent\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658461    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658490    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-kubelet\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658507    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-run-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658515    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-netns\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658542    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-netd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658568    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-script-lib\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658594    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-sys-fs\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.660224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658610    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-iptables-alerter-script\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658623    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-conf\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658460    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50f2f01d-8199-4e84-ab1b-96c4a1a06803-cni-binary-copy\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658619    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-tuned\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658691    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-sys\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658699    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-os-release\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658693    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cnibin\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658703    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-host-run-multus-certs\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658730    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cnibin\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658747    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-slash\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658773    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-bin\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658798    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-config\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658801    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-sysctl-d\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658824    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-os-release\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658845    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-device-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658850    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-system-cni-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658875    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-var-lib-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.660740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658902    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovn-node-metrics-cert\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658927    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658943    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-sys-fs\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6197213-20ec-4de9-8308-3f48b0cee933-hosts-file\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.658983    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659062    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659103    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-slash\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659105    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b405a8c5-b718-4ec6-bebe-2f1209fe5859-cni-binary-copy\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659140    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b405a8c5-b718-4ec6-bebe-2f1209fe5859-system-cni-dir\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659153    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-var-lib-openvswitch\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659178    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-bin\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659353    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-script-lib\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659442    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-kubelet\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659489    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-run-netns\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659534    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-host-cni-netd\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659542    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50f2f01d-8199-4e84-ab1b-96c4a1a06803-os-release\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/961fc510-7c65-49b2-a4b3-b193252a6268-kubelet-dir\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.661242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.659640    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovnkube-config\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.661860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.660336    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c466082b-97a7-404a-91c0-8c9ac1e9c791-agent-certs\") pod \"konnectivity-agent-bxkxv\" (UID: \"c466082b-97a7-404a-91c0-8c9ac1e9c791\") " pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.661860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.660573    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-tmp\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.661860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.661002    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-etc-tuned\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.661860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.661759    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-ovn-node-metrics-cert\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.668452 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.667811    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:06.668452 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.667839    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:06.668452 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.667854    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:06.668452 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:06.667925    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:07.167904539 +0000 UTC m=+3.128452008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:06.670434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.670388    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbm4\" (UniqueName: \"kubernetes.io/projected/50f2f01d-8199-4e84-ab1b-96c4a1a06803-kube-api-access-mkbm4\") pod \"multus-cjv66\" (UID: \"50f2f01d-8199-4e84-ab1b-96c4a1a06803\") " pod="openshift-multus/multus-cjv66"
May 26 11:25:06.670538 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.670452    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9d5r\" (UniqueName: \"kubernetes.io/projected/b405a8c5-b718-4ec6-bebe-2f1209fe5859-kube-api-access-w9d5r\") pod \"multus-additional-cni-plugins-z2bzc\" (UID: \"b405a8c5-b718-4ec6-bebe-2f1209fe5859\") " pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.671142 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.671120    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rszt\" (UniqueName: \"kubernetes.io/projected/5510eac5-cefa-4f2a-9407-70a4e42cd9e3-kube-api-access-6rszt\") pod \"iptables-alerter-h4pkj\" (UID: \"5510eac5-cefa-4f2a-9407-70a4e42cd9e3\") " pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.671284 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.671264    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mgw\" (UniqueName: \"kubernetes.io/projected/6d7c4ee1-df53-4ffa-b5f1-9bf483055877-kube-api-access-p6mgw\") pod \"tuned-pjhw9\" (UID: \"6d7c4ee1-df53-4ffa-b5f1-9bf483055877\") " pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.671356 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.671286    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfn9m\" (UniqueName: \"kubernetes.io/projected/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-kube-api-access-hfn9m\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:06.671356 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.671270    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvl6w\" (UniqueName: \"kubernetes.io/projected/961fc510-7c65-49b2-a4b3-b193252a6268-kube-api-access-vvl6w\") pod \"aws-ebs-csi-driver-node-59qdl\" (UID: \"961fc510-7c65-49b2-a4b3-b193252a6268\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.671356 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.671272    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwv2t\" (UniqueName: \"kubernetes.io/projected/fb187e2a-b37a-48ad-8f06-2f74b6bc867e-kube-api-access-fwv2t\") pod \"ovnkube-node-b8trm\" (UID: \"fb187e2a-b37a-48ad-8f06-2f74b6bc867e\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.760174 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760134    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6197213-20ec-4de9-8308-3f48b0cee933-hosts-file\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760227    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb390ce-0700-4745-9dda-fe590c8284c2-host\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760265    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6197213-20ec-4de9-8308-3f48b0cee933-tmp-dir\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760295    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdxz\" (UniqueName: \"kubernetes.io/projected/c6197213-20ec-4de9-8308-3f48b0cee933-kube-api-access-ksdxz\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cb390ce-0700-4745-9dda-fe590c8284c2-host\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760324    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cb390ce-0700-4745-9dda-fe590c8284c2-serviceca\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.760353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760259    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6197213-20ec-4de9-8308-3f48b0cee933-hosts-file\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.760669 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760400    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wf4\" (UniqueName: \"kubernetes.io/projected/5cb390ce-0700-4745-9dda-fe590c8284c2-kube-api-access-q6wf4\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.760727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760695    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6197213-20ec-4de9-8308-3f48b0cee933-tmp-dir\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.760774 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.760759    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cb390ce-0700-4745-9dda-fe590c8284c2-serviceca\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.769200 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.769127    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wf4\" (UniqueName: \"kubernetes.io/projected/5cb390ce-0700-4745-9dda-fe590c8284c2-kube-api-access-q6wf4\") pod \"node-ca-f8m2x\" (UID: \"5cb390ce-0700-4745-9dda-fe590c8284c2\") " pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:06.769200 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.769147    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdxz\" (UniqueName: \"kubernetes.io/projected/c6197213-20ec-4de9-8308-3f48b0cee933-kube-api-access-ksdxz\") pod \"node-resolver-lsn6g\" (UID: \"c6197213-20ec-4de9-8308-3f48b0cee933\") " pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.774315 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.774292    2562 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:06.784899 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.784874    2562 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:25:06.836759 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.836729    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h4pkj"
May 26 11:25:06.843721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.843694    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cjv66"
May 26 11:25:06.853450 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.853408    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pjhw9"
May 26 11:25:06.860109 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.860082    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z2bzc"
May 26 11:25:06.866873 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.866850    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:06.874488 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.874464    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:06.881015 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.880991    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl"
May 26 11:25:06.887727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.887708    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lsn6g"
May 26 11:25:06.892299 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:06.892279    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8m2x"
May 26 11:25:07.162498 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.162442    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:07.162637 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.162581    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:07.162678 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.162642    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:08.162627318 +0000 UTC m=+4.123174798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:07.203643 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.203619    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb405a8c5_b718_4ec6_bebe_2f1209fe5859.slice/crio-6defa524760de47274023660bcdff39ae3ac1eea51efecf6ab43c480f9be8af4 WatchSource:0}: Error finding container 6defa524760de47274023660bcdff39ae3ac1eea51efecf6ab43c480f9be8af4: Status 404 returned error can't find the container with id 6defa524760de47274023660bcdff39ae3ac1eea51efecf6ab43c480f9be8af4
May 26 11:25:07.204873 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.204801    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5510eac5_cefa_4f2a_9407_70a4e42cd9e3.slice/crio-523b84524b83354f41a1038ab0308ec4060d40d43cca3b328d26f6dae0fb976d WatchSource:0}: Error finding container 523b84524b83354f41a1038ab0308ec4060d40d43cca3b328d26f6dae0fb976d: Status 404 returned error can't find the container with id 523b84524b83354f41a1038ab0308ec4060d40d43cca3b328d26f6dae0fb976d
May 26 11:25:07.205505 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.205478    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f2f01d_8199_4e84_ab1b_96c4a1a06803.slice/crio-e39ca800b2070f82518c61eb49bda2f579cbd3143f2f1737160ef7a6cfe9d53e WatchSource:0}: Error finding container e39ca800b2070f82518c61eb49bda2f579cbd3143f2f1737160ef7a6cfe9d53e: Status 404 returned error can't find the container with id e39ca800b2070f82518c61eb49bda2f579cbd3143f2f1737160ef7a6cfe9d53e
May 26 11:25:07.207230 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.207201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7c4ee1_df53_4ffa_b5f1_9bf483055877.slice/crio-476eeb0158704f854fda29dac282038918962ec4502bae80e8c411cbb66ab869 WatchSource:0}: Error finding container 476eeb0158704f854fda29dac282038918962ec4502bae80e8c411cbb66ab869: Status 404 returned error can't find the container with id 476eeb0158704f854fda29dac282038918962ec4502bae80e8c411cbb66ab869
May 26 11:25:07.208460 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.208172    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc466082b_97a7_404a_91c0_8c9ac1e9c791.slice/crio-16373a939b8cf53aeb1a2936632c34a0db2525723018b38c8d6a50e10f55e020 WatchSource:0}: Error finding container 16373a939b8cf53aeb1a2936632c34a0db2525723018b38c8d6a50e10f55e020: Status 404 returned error can't find the container with id 16373a939b8cf53aeb1a2936632c34a0db2525723018b38c8d6a50e10f55e020
May 26 11:25:07.209162 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.209140    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961fc510_7c65_49b2_a4b3_b193252a6268.slice/crio-7741c2adfcb03b926cc13889aca91e50cb76efaa83b1a0a7fc357b656d312163 WatchSource:0}: Error finding container 7741c2adfcb03b926cc13889aca91e50cb76efaa83b1a0a7fc357b656d312163: Status 404 returned error can't find the container with id 7741c2adfcb03b926cc13889aca91e50cb76efaa83b1a0a7fc357b656d312163
May 26 11:25:07.210849 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.210798    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb390ce_0700_4745_9dda_fe590c8284c2.slice/crio-cbb4654860da1046e3e169aa61ea2de1308369e219ccaa438fd0d4f85a7e48ac WatchSource:0}: Error finding container cbb4654860da1046e3e169aa61ea2de1308369e219ccaa438fd0d4f85a7e48ac: Status 404 returned error can't find the container with id cbb4654860da1046e3e169aa61ea2de1308369e219ccaa438fd0d4f85a7e48ac
May 26 11:25:07.212083 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.212060    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb187e2a_b37a_48ad_8f06_2f74b6bc867e.slice/crio-5b553133c3ac548fdea13458e5c6125a22c59ec4aff38efbaf562bc36678d1f5 WatchSource:0}: Error finding container 5b553133c3ac548fdea13458e5c6125a22c59ec4aff38efbaf562bc36678d1f5: Status 404 returned error can't find the container with id 5b553133c3ac548fdea13458e5c6125a22c59ec4aff38efbaf562bc36678d1f5
May 26 11:25:07.213754 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:07.213634    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6197213_20ec_4de9_8308_3f48b0cee933.slice/crio-7c967c7636bb925405b28c22f460193e7832c62e9f00dd1e065780bddd1adffb WatchSource:0}: Error finding container 7c967c7636bb925405b28c22f460193e7832c62e9f00dd1e065780bddd1adffb: Status 404 returned error can't find the container with id 7c967c7636bb925405b28c22f460193e7832c62e9f00dd1e065780bddd1adffb
May 26 11:25:07.263147 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.263119    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:07.263279 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.263260    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:07.263314 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.263283    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:07.263314 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.263293    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:07.263391 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.263340    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:08.263327204 +0000 UTC m=+4.223874666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:07.585270 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.585015    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:20:05 +0000 UTC" deadline="2028-03-01 17:30:27.665816622 +0000 UTC"
May 26 11:25:07.585270 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.585269    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15486h5m20.080552733s"
May 26 11:25:07.674591 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.674055    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:07.674591 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:07.674201    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:07.684145 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.684079    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerStarted","Data":"6defa524760de47274023660bcdff39ae3ac1eea51efecf6ab43c480f9be8af4"}
May 26 11:25:07.688653 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.687974    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"f88796f630f0b1df3d5abb08c2fec007","Type":"ContainerStarted","Data":"ca5c60c52ed23ed70a1b5b58ea2d00efb1d4bf75ee55408aa3043ede26339b2c"}
May 26 11:25:07.690220 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.690156    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lsn6g" event={"ID":"c6197213-20ec-4de9-8308-3f48b0cee933","Type":"ContainerStarted","Data":"7c967c7636bb925405b28c22f460193e7832c62e9f00dd1e065780bddd1adffb"}
May 26 11:25:07.692766 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.692723    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"5b553133c3ac548fdea13458e5c6125a22c59ec4aff38efbaf562bc36678d1f5"}
May 26 11:25:07.695428 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.695393    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjv66" event={"ID":"50f2f01d-8199-4e84-ab1b-96c4a1a06803","Type":"ContainerStarted","Data":"e39ca800b2070f82518c61eb49bda2f579cbd3143f2f1737160ef7a6cfe9d53e"}
May 26 11:25:07.701805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.701780    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl" event={"ID":"961fc510-7c65-49b2-a4b3-b193252a6268","Type":"ContainerStarted","Data":"7741c2adfcb03b926cc13889aca91e50cb76efaa83b1a0a7fc357b656d312163"}
May 26 11:25:07.714033 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.714004    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pjhw9" event={"ID":"6d7c4ee1-df53-4ffa-b5f1-9bf483055877","Type":"ContainerStarted","Data":"476eeb0158704f854fda29dac282038918962ec4502bae80e8c411cbb66ab869"}
May 26 11:25:07.721001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.720971    2562 generic.go:358] "Generic (PLEG): container finished" podID="839c040c48a39333914f45b99d70dd25" containerID="4f587c296bc62b2a1c20eacaf8785fdbe522cc014213cc80d6270caa559baee0" exitCode=0
May 26 11:25:07.721360 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.721317    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"839c040c48a39333914f45b99d70dd25","Type":"ContainerDied","Data":"4f587c296bc62b2a1c20eacaf8785fdbe522cc014213cc80d6270caa559baee0"}
May 26 11:25:07.733569 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.733494    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8m2x" event={"ID":"5cb390ce-0700-4745-9dda-fe590c8284c2","Type":"ContainerStarted","Data":"cbb4654860da1046e3e169aa61ea2de1308369e219ccaa438fd0d4f85a7e48ac"}
May 26 11:25:07.735380 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.735344    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxkxv" event={"ID":"c466082b-97a7-404a-91c0-8c9ac1e9c791","Type":"ContainerStarted","Data":"16373a939b8cf53aeb1a2936632c34a0db2525723018b38c8d6a50e10f55e020"}
May 26 11:25:07.736484 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.736119    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" podStartSLOduration=2.73610229 podStartE2EDuration="2.73610229s" podCreationTimestamp="2026-05-26 11:25:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:07.703672785 +0000 UTC m=+3.664220271" watchObservedRunningTime="2026-05-26 11:25:07.73610229 +0000 UTC m=+3.696649777"
May 26 11:25:07.748668 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:07.746888    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h4pkj" event={"ID":"5510eac5-cefa-4f2a-9407-70a4e42cd9e3","Type":"ContainerStarted","Data":"523b84524b83354f41a1038ab0308ec4060d40d43cca3b328d26f6dae0fb976d"}
May 26 11:25:08.170516 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:08.169954    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:08.170516 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.170091    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:08.170516 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.170155    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:10.170136967 +0000 UTC m=+6.130684436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:08.272864 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:08.272184    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:08.272864 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.272392    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:08.272864 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.272409    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:08.272864 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.272441    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:08.272864 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.272499    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:10.272480502 +0000 UTC m=+6.233027972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:08.674231 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:08.674196    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:08.674706 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:08.674343    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:08.771999 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:08.771965    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"839c040c48a39333914f45b99d70dd25","Type":"ContainerStarted","Data":"4f9f4f7f752592ed63962219edb5b501c66299a487e5a79e564ad580e56312e3"}
May 26 11:25:09.674846 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:09.674812    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:09.675320 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:09.674951    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:10.191471 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:10.191398    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:10.191650 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.191571    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:10.191650 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.191636    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.191617906 +0000 UTC m=+10.152165375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:10.292303 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:10.292265    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:10.292494 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.292476    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:10.292562 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.292504    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:10.292562 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.292519    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:10.292667 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.292588    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.292568209 +0000 UTC m=+10.253115686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:10.675237 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:10.675206    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:10.675687 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:10.675352    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:11.674427 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:11.673913    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:11.674427 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:11.674042    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:12.681583 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:12.681216    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:12.681583 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:12.681388    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:13.674459 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:13.674426    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:13.674625 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:13.674552    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:14.225458 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:14.225406    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:14.225925 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.225553    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:14.225925 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.225627    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:22.225606448 +0000 UTC m=+18.186153916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:14.326355 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:14.326320    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:14.326547 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.326512    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:14.326547 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.326530    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:14.326547 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.326541    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:14.326699 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.326597    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:22.32657971 +0000 UTC m=+18.287127188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:14.675669 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:14.674996    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:14.675669 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:14.675117    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:15.673906 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:15.673871    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:15.674344 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:15.674009    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:16.673916 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:16.673883    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:16.674371 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:16.674018    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:17.674507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:17.674472    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:17.674946 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:17.674581    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:18.677320 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:18.677282    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:18.677797 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:18.677433    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:19.674326 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:19.674295    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:19.674521 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:19.674433    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:20.674297 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:20.674259    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:20.674724 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:20.674428    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:21.674923 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:21.674890    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:21.675351 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:21.675007    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:22.281201 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:22.281161    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:22.281377 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.281320    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:22.281467 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.281394    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:38.281372814 +0000 UTC m=+34.241920282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:22.381684 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:22.381649    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:22.381862 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.381783    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:22.381862 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.381798    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:22.381862 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.381809    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:22.381992 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.381866    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:25:38.381851014 +0000 UTC m=+34.342398477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:22.674241 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:22.674202    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:22.674403 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:22.674337    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:23.674366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:23.674326    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:23.674781 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:23.674468    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:24.678280 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:24.678255    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:24.678698 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:24.678345    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:25.674190 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.674005    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:25.674285 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:25.674267    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:25.802064 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.802028    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl" event={"ID":"961fc510-7c65-49b2-a4b3-b193252a6268","Type":"ContainerStarted","Data":"f09f7273e31226f7ae9a9289cb85657e9907095157967153582744d97e120893"}
May 26 11:25:25.803959 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.803936    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pjhw9" event={"ID":"6d7c4ee1-df53-4ffa-b5f1-9bf483055877","Type":"ContainerStarted","Data":"bbf36859203529f800dd1abf0b32457a185d452e3e4ee23b2d4feeebc5c20599"}
May 26 11:25:25.805470 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.805445    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8m2x" event={"ID":"5cb390ce-0700-4745-9dda-fe590c8284c2","Type":"ContainerStarted","Data":"ba480f32565966a6e5b6a56c4da121e9e5ba10818ba03234c98fdebbdfcee2e1"}
May 26 11:25:25.806850 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.806830    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxkxv" event={"ID":"c466082b-97a7-404a-91c0-8c9ac1e9c791","Type":"ContainerStarted","Data":"07c61ad29fccfaa2e89e085b7c927c90660f7f2e3c9efbe25dcf0f24f8f4b5a2"}
May 26 11:25:25.812741 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.812654    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="16a3fdc8cfee13cf4db897383c93be080beb61d4bcd7208a45b17dd6fc7ac0b0" exitCode=0
May 26 11:25:25.812833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.812788    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"16a3fdc8cfee13cf4db897383c93be080beb61d4bcd7208a45b17dd6fc7ac0b0"}
May 26 11:25:25.815124 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.815101    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lsn6g" event={"ID":"c6197213-20ec-4de9-8308-3f48b0cee933","Type":"ContainerStarted","Data":"bace708edc2d3d4e7646027e8306d4179d70b7f96ba5a2f91fb0a0dfb1cff384"}
May 26 11:25:25.817971 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.817956    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:25:25.818355 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.818332    2562 generic.go:358] "Generic (PLEG): container finished" podID="fb187e2a-b37a-48ad-8f06-2f74b6bc867e" containerID="9015ae7747dbdf3d666f71631f13275e26b1929f787ea395d63ff7dadc9f34ee" exitCode=1
May 26 11:25:25.818483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.818398    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"38f787d4fdb484f81334e61a1072e7bbaa37935ce46d5dfe796e6b4cf1906aa7"}
May 26 11:25:25.818483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.818439    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"1cf3f6b913f74bb1208753af587bd7407a63be74c0c92a9f705fcbd812db4fd2"}
May 26 11:25:25.818483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.818454    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerDied","Data":"9015ae7747dbdf3d666f71631f13275e26b1929f787ea395d63ff7dadc9f34ee"}
May 26 11:25:25.818483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.818468    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"331388b16e7fefc463272fc6a1bd756259fa69bcc60b669668ce08de7f1981d2"}
May 26 11:25:25.819787 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.819764    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cjv66" event={"ID":"50f2f01d-8199-4e84-ab1b-96c4a1a06803","Type":"ContainerStarted","Data":"af58b4df3608caca9608980293dd84783f5036faa22615ed89a89a5097ea0bf4"}
May 26 11:25:25.827852 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.827810    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pjhw9" podStartSLOduration=3.942215455 podStartE2EDuration="21.827796733s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.209846462 +0000 UTC m=+3.170393938" lastFinishedPulling="2026-05-26 11:25:25.095427738 +0000 UTC m=+21.055975216" observedRunningTime="2026-05-26 11:25:25.827643397 +0000 UTC m=+21.788190875" watchObservedRunningTime="2026-05-26 11:25:25.827796733 +0000 UTC m=+21.788344260"
May 26 11:25:25.827941 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.827918    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" podStartSLOduration=20.827914237 podStartE2EDuration="20.827914237s" podCreationTimestamp="2026-05-26 11:25:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:25:08.793313782 +0000 UTC m=+4.753861269" watchObservedRunningTime="2026-05-26 11:25:25.827914237 +0000 UTC m=+21.788461722"
May 26 11:25:25.867580 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.867531    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bxkxv" podStartSLOduration=8.613936487 podStartE2EDuration="21.867517202s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.210143129 +0000 UTC m=+3.170690593" lastFinishedPulling="2026-05-26 11:25:20.463723841 +0000 UTC m=+16.424271308" observedRunningTime="2026-05-26 11:25:25.866965811 +0000 UTC m=+21.827513294" watchObservedRunningTime="2026-05-26 11:25:25.867517202 +0000 UTC m=+21.828064689"
May 26 11:25:25.892188 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.892151    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-lsn6g" podStartSLOduration=4.050392028 podStartE2EDuration="21.892138676s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.215478114 +0000 UTC m=+3.176025578" lastFinishedPulling="2026-05-26 11:25:25.057224746 +0000 UTC m=+21.017772226" observedRunningTime="2026-05-26 11:25:25.891752382 +0000 UTC m=+21.852299866" watchObservedRunningTime="2026-05-26 11:25:25.892138676 +0000 UTC m=+21.852686161"
May 26 11:25:25.926815 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.926777    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-cjv66" podStartSLOduration=4.017515029 podStartE2EDuration="21.926764779s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.207289181 +0000 UTC m=+3.167836644" lastFinishedPulling="2026-05-26 11:25:25.116538925 +0000 UTC m=+21.077086394" observedRunningTime="2026-05-26 11:25:25.926684164 +0000 UTC m=+21.887231667" watchObservedRunningTime="2026-05-26 11:25:25.926764779 +0000 UTC m=+21.887312264"
May 26 11:25:25.927272 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:25.927248    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f8m2x" podStartSLOduration=4.048896867 podStartE2EDuration="21.927243018s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.21331035 +0000 UTC m=+3.173857818" lastFinishedPulling="2026-05-26 11:25:25.09165649 +0000 UTC m=+21.052203969" observedRunningTime="2026-05-26 11:25:25.907375165 +0000 UTC m=+21.867922651" watchObservedRunningTime="2026-05-26 11:25:25.927243018 +0000 UTC m=+21.887790503"
May 26 11:25:26.322756 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.322730    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:25:26.524539 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.524460    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:26.525308 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.525283    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:26.629702 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.629609    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:25:26.322753357Z","UUID":"069dc10a-d53b-41a4-a388-196d8457d118","Handler":null,"Name":"","Endpoint":""}
May 26 11:25:26.631984 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.631950    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:25:26.632143 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.631995    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:25:26.676569 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.676542    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:26.676728 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:26.676647    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:26.826965 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.826873    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h4pkj" event={"ID":"5510eac5-cefa-4f2a-9407-70a4e42cd9e3","Type":"ContainerStarted","Data":"941658fb0bf2baa07ef0c2715752c46e4d9675de1400a480e33f735d942c540f"}
May 26 11:25:26.831062 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.831041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:25:26.831447 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.831405    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"3b81580093571fc2599c264ede14ef494db928679d7ae46973e0d49665849f0d"}
May 26 11:25:26.831608 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.831453    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"aa7e160b20e6ac344996e030f73e6f41f618f57557b4ad770d7c73ab93ee4002"}
May 26 11:25:26.833571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.833437    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl" event={"ID":"961fc510-7c65-49b2-a4b3-b193252a6268","Type":"ContainerStarted","Data":"b6dfb21c5c5a48f98ce8e9ecd1b4ebefeff398519b848b15210d8b296a94972d"}
May 26 11:25:26.833865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.833845    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:26.834432 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.834393    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bxkxv"
May 26 11:25:26.846074 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:26.846025    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h4pkj" podStartSLOduration=4.9609934970000005 podStartE2EDuration="22.846009905s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.206552764 +0000 UTC m=+3.167100237" lastFinishedPulling="2026-05-26 11:25:25.091569167 +0000 UTC m=+21.052116645" observedRunningTime="2026-05-26 11:25:26.84584444 +0000 UTC m=+22.806391925" watchObservedRunningTime="2026-05-26 11:25:26.846009905 +0000 UTC m=+22.806557392"
May 26 11:25:27.674622 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:27.674538    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:27.674778 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:27.674653    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:27.839147 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:27.839085    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl" event={"ID":"961fc510-7c65-49b2-a4b3-b193252a6268","Type":"ContainerStarted","Data":"6de25b9071320ce099a5db97d200ec901370c07c77f8dea830864a8a49f98563"}
May 26 11:25:27.863867 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:27.863801    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-59qdl" podStartSLOduration=3.694404301 podStartE2EDuration="23.863785271s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.211065839 +0000 UTC m=+3.171613308" lastFinishedPulling="2026-05-26 11:25:27.3804468 +0000 UTC m=+23.340994278" observedRunningTime="2026-05-26 11:25:27.86315809 +0000 UTC m=+23.823705574" watchObservedRunningTime="2026-05-26 11:25:27.863785271 +0000 UTC m=+23.824332768"
May 26 11:25:28.674991 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:28.674953    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:28.675141 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:28.675113    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:28.844239 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:28.844215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:25:28.844709 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:28.844666    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"deabddd38657380a3fd1f65f002ad2a586857f4d57085bf85d8905822e29814b"}
May 26 11:25:29.674561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:29.674325    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:29.674733 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:29.674666    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:30.673990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.673953    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:30.674695 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:30.674095    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:30.848942 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.848907    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="4ce7fa71ce528c5c4a459659b828aea6b9edc3340bba6074735a1c84d4aa0040" exitCode=0
May 26 11:25:30.849097 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.848994    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"4ce7fa71ce528c5c4a459659b828aea6b9edc3340bba6074735a1c84d4aa0040"}
May 26 11:25:30.852024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.851874    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:25:30.852283 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.852264    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"fadd42588c49994ac040affb5e0adff881300b9e9f49de865bb12f321d7be1c7"}
May 26 11:25:30.852618 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.852605    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:30.852679 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.852622    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:30.852756 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.852738    2562 scope.go:117] "RemoveContainer" containerID="9015ae7747dbdf3d666f71631f13275e26b1929f787ea395d63ff7dadc9f34ee"
May 26 11:25:30.868481 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.868460    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:30.868566 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:30.868529    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:31.674862 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:31.674830    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:31.675300 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:31.674945    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:31.861033 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:31.861003    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:25:31.861394 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:31.861364    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" event={"ID":"fb187e2a-b37a-48ad-8f06-2f74b6bc867e","Type":"ContainerStarted","Data":"c7d35bd34b3bd3ee0defe5498c9cc8c9c9fd6f170f98bf10edd794e36d4415f6"}
May 26 11:25:31.861594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:31.861573    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 11:25:31.897991 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:31.897936    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm" podStartSLOduration=9.969348295 podStartE2EDuration="27.897921898s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.214086138 +0000 UTC m=+3.174633600" lastFinishedPulling="2026-05-26 11:25:25.142659729 +0000 UTC m=+21.103207203" observedRunningTime="2026-05-26 11:25:31.897186509 +0000 UTC m=+27.857734030" watchObservedRunningTime="2026-05-26 11:25:31.897921898 +0000 UTC m=+27.858469383"
May 26 11:25:32.130747 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.130718    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hkf6m"]
May 26 11:25:32.130881 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.130869    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:32.130983 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:32.130964    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:32.136362 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.136338    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6866p"]
May 26 11:25:32.136487 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.136474    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:32.136592 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:32.136568    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:32.864924 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.864888    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="a670a5be5e4e984b06107003492f89736c300dfb1c194d882682c8a2748be5d6" exitCode=0
May 26 11:25:32.865436 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.864975    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"a670a5be5e4e984b06107003492f89736c300dfb1c194d882682c8a2748be5d6"}
May 26 11:25:32.865436 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:32.865078    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 11:25:33.674457 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:33.674403    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:33.674640 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:33.674404    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:33.674640 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:33.674545    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:33.674640 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:33.674605    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:34.764201 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:34.764030    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:25:34.871990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:34.871963    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="e808a2e447f69561bbca6973f2b3c8aa257f12c2b57448d864f3a9b8ee00f328" exitCode=0
May 26 11:25:34.872103 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:34.872001    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"e808a2e447f69561bbca6973f2b3c8aa257f12c2b57448d864f3a9b8ee00f328"}
May 26 11:25:35.674599 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:35.674562    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:35.674789 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:35.674571    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:35.674789 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:35.674672    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:35.674900 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:35.674798    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:37.674680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:37.674645    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:37.675337 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:37.674753    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6866p" podUID="659bc118-373e-494f-a4f4-e6deaeea6fed"
May 26 11:25:37.675337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:37.674824    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:37.675337 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:37.674959    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:25:38.303590 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.303554    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:38.303782 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.303668    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:38.303782 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.303733    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:10.303712608 +0000 UTC m=+66.264260074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:38.404117 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.404078    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:38.404334 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.404254    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:38.404334 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.404273    2562 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:38.404334 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.404286    2562 projected.go:194] Error preparing data for projected volume kube-api-access-99p8d for pod openshift-network-diagnostics/network-check-target-6866p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:38.404518 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:38.404353    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d podName:659bc118-373e-494f-a4f4-e6deaeea6fed nodeName:}" failed. No retries permitted until 2026-05-26 11:26:10.404334995 +0000 UTC m=+66.364882457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-99p8d" (UniqueName: "kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d") pod "network-check-target-6866p" (UID: "659bc118-373e-494f-a4f4-e6deaeea6fed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:38.878336 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.878305    2562 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeReady"
May 26 11:25:38.878928 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.878460    2562 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:25:38.956140 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.956062    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7t"]
May 26 11:25:38.996391 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.996352    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6ntbs"]
May 26 11:25:38.996576 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:38.996544    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.001457 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.001433    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:25:39.002016 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.001997    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:25:39.002134 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.002020    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:25:39.002322 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.002304    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwmqw\""
May 26 11:25:39.015755 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.015734    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7t"]
May 26 11:25:39.015844 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.015762    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ntbs"]
May 26 11:25:39.015904 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.015869    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.020458 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.020437    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:25:39.020574 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.020473    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:25:39.020574 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.020523    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6qhf\""
May 26 11:25:39.110089 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lh8j\" (UniqueName: \"kubernetes.io/projected/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-kube-api-access-2lh8j\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.110089 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b140c9d9-dd67-4b37-9745-557d148045d9-config-volume\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.110337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b140c9d9-dd67-4b37-9745-557d148045d9-tmp-dir\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.110337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.110337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.110337 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.110290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxh7\" (UniqueName: \"kubernetes.io/projected/b140c9d9-dd67-4b37-9745-557d148045d9-kube-api-access-xqxh7\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211113 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211019    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211284 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211085    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxh7\" (UniqueName: \"kubernetes.io/projected/b140c9d9-dd67-4b37-9745-557d148045d9-kube-api-access-xqxh7\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211284 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.211192    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:39.211284 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211215    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2lh8j\" (UniqueName: \"kubernetes.io/projected/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-kube-api-access-2lh8j\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.211284 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211246    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b140c9d9-dd67-4b37-9745-557d148045d9-config-volume\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211284 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.211267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:39.711247051 +0000 UTC m=+35.671794532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:39.211586 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211325    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b140c9d9-dd67-4b37-9745-557d148045d9-tmp-dir\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211586 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211365    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.211586 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.211494    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:39.211586 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.211540    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:39.711524607 +0000 UTC m=+35.672072076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:25:39.211737 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211658    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b140c9d9-dd67-4b37-9745-557d148045d9-tmp-dir\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.211810 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.211779    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b140c9d9-dd67-4b37-9745-557d148045d9-config-volume\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.230631 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.230603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lh8j\" (UniqueName: \"kubernetes.io/projected/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-kube-api-access-2lh8j\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.230808 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.230787    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxh7\" (UniqueName: \"kubernetes.io/projected/b140c9d9-dd67-4b37-9745-557d148045d9-kube-api-access-xqxh7\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.674849 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.674813    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:25:39.674849 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.674843    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:25:39.680690 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.680671    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:39.680823 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.680735    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:39.681348 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.681326    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf2lb\""
May 26 11:25:39.681472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.681333    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:39.681472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.681387    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gx5q\""
May 26 11:25:39.718846 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.718824    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:39.718973 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:39.718862    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:39.719036 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.718984    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:39.719072 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.719035    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:39.719072 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.719067    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:40.719046457 +0000 UTC m=+36.679593939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:25:39.719158 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:39.719084    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:40.719075535 +0000 UTC m=+36.679622997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:40.726327 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:40.726299    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:40.728182 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:40.726344    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:40.728182 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:40.726507    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:40.728182 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:40.726520    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:40.728182 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:40.726576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:42.726554527 +0000 UTC m=+38.687102005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:25:40.728182 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:40.726643    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:42.726626886 +0000 UTC m=+38.687174349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:40.886347 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:40.886319    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="b5c6a0b7247b8653550422f69eeb8012519387255ef1f8e2a44c01b9d96aba52" exitCode=0
May 26 11:25:40.886503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:40.886368    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"b5c6a0b7247b8653550422f69eeb8012519387255ef1f8e2a44c01b9d96aba52"}
May 26 11:25:41.890888 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:41.890852    2562 generic.go:358] "Generic (PLEG): container finished" podID="b405a8c5-b718-4ec6-bebe-2f1209fe5859" containerID="b3c9424addb87d54cc32c2887eefb630473a29792afbf4d2ef62c3be3fae2427" exitCode=0
May 26 11:25:41.891251 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:41.890903    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerDied","Data":"b3c9424addb87d54cc32c2887eefb630473a29792afbf4d2ef62c3be3fae2427"}
May 26 11:25:42.745129 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:42.745080    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:42.745317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:42.745170    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:42.745317 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:42.745235    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:42.745317 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:42.745246    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:42.745317 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:42.745294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:46.74528009 +0000 UTC m=+42.705827567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:25:42.745317 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:42.745307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:46.745301014 +0000 UTC m=+42.705848477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:42.895471 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:42.895436    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" event={"ID":"b405a8c5-b718-4ec6-bebe-2f1209fe5859","Type":"ContainerStarted","Data":"f273c77d85e3bd433b26745af21764cf8313e2280bfdcda563fe4dee0c040ade"}
May 26 11:25:42.939727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:42.939674    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z2bzc" podStartSLOduration=5.626737063 podStartE2EDuration="38.939662599s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:25:07.205239843 +0000 UTC m=+3.165787306" lastFinishedPulling="2026-05-26 11:25:40.518165372 +0000 UTC m=+36.478712842" observedRunningTime="2026-05-26 11:25:42.939488819 +0000 UTC m=+38.900036305" watchObservedRunningTime="2026-05-26 11:25:42.939662599 +0000 UTC m=+38.900210084"
May 26 11:25:46.770119 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:46.770065    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:46.770513 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:46.770144    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:46.770513 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:46.770208    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:46.770513 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:46.770272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:54.770256474 +0000 UTC m=+50.730803949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:25:46.770513 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:46.770284    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:46.770513 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:46.770337    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:54.770320915 +0000 UTC m=+50.730868378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:52.757364 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.757330    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"]
May 26 11:25:52.781191 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.781166    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.783010 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.782830    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"]
May 26 11:25:52.785925 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.785905    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 26 11:25:52.786479 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.786460    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 26 11:25:52.786902 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.786884    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 26 11:25:52.786902 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.786894    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 26 11:25:52.806853 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.806831    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"]
May 26 11:25:52.810939 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.810916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f918f-9d67-4933-a823-3d12b2f2da74-tmp\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.811057 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.810967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqj4\" (UniqueName: \"kubernetes.io/projected/3b6f918f-9d67-4933-a823-3d12b2f2da74-kube-api-access-vvqj4\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.811057 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.811009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3b6f918f-9d67-4933-a823-3d12b2f2da74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.830975 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.830952    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.835148 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.835121    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 26 11:25:52.835148 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.835123    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 26 11:25:52.836383 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.836343    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 26 11:25:52.836569 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.836553    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 26 11:25:52.838928 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.838908    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"]
May 26 11:25:52.911819 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdj4\" (UniqueName: \"kubernetes.io/projected/9219ee55-493b-4526-acec-a9c39193d91f-kube-api-access-kcdj4\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911830    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqj4\" (UniqueName: \"kubernetes.io/projected/3b6f918f-9d67-4933-a823-3d12b2f2da74-kube-api-access-vvqj4\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911873    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3b6f918f-9d67-4933-a823-3d12b2f2da74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9219ee55-493b-4526-acec-a9c39193d91f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912024 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.911995    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f918f-9d67-4933-a823-3d12b2f2da74-tmp\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.912306 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.912079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912306 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.912117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:52.912442 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.912401    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f918f-9d67-4933-a823-3d12b2f2da74-tmp\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.915262 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.915238    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3b6f918f-9d67-4933-a823-3d12b2f2da74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:52.920402 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:52.920383    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqj4\" (UniqueName: \"kubernetes.io/projected/3b6f918f-9d67-4933-a823-3d12b2f2da74-kube-api-access-vvqj4\") pod \"klusterlet-addon-workmgr-8c7ff64f6-vzrsx\" (UID: \"3b6f918f-9d67-4933-a823-3d12b2f2da74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:53.012608 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012538    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.012608 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012575    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.012608 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012599    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdj4\" (UniqueName: \"kubernetes.io/projected/9219ee55-493b-4526-acec-a9c39193d91f-kube-api-access-kcdj4\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.012818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012657    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.012818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012701    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9219ee55-493b-4526-acec-a9c39193d91f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.012818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.012733    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.013491 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.013467    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/9219ee55-493b-4526-acec-a9c39193d91f-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.015003 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.014975    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.015116 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.015095    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.015363 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.015346    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-ca\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.015452 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.015433    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9219ee55-493b-4526-acec-a9c39193d91f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.021543 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.021515    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdj4\" (UniqueName: \"kubernetes.io/projected/9219ee55-493b-4526-acec-a9c39193d91f-kube-api-access-kcdj4\") pod \"cluster-proxy-proxy-agent-c4788496-qfhzj\" (UID: \"9219ee55-493b-4526-acec-a9c39193d91f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.090985 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.090955    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:25:53.152309 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.152275    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:25:53.253940 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.253913    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"]
May 26 11:25:53.257378 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:53.257349    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6f918f_9d67_4933_a823_3d12b2f2da74.slice/crio-482536f6e9bd3f90be4f669ed952ad371dcf9d76f61c2274f300999c6870003d WatchSource:0}: Error finding container 482536f6e9bd3f90be4f669ed952ad371dcf9d76f61c2274f300999c6870003d: Status 404 returned error can't find the container with id 482536f6e9bd3f90be4f669ed952ad371dcf9d76f61c2274f300999c6870003d
May 26 11:25:53.290882 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.290809    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"]
May 26 11:25:53.294237 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:25:53.294212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9219ee55_493b_4526_acec_a9c39193d91f.slice/crio-0bc619175f298d9d08068f98daef306c11032a31a24e003a0609af5afa023ab8 WatchSource:0}: Error finding container 0bc619175f298d9d08068f98daef306c11032a31a24e003a0609af5afa023ab8: Status 404 returned error can't find the container with id 0bc619175f298d9d08068f98daef306c11032a31a24e003a0609af5afa023ab8
May 26 11:25:53.915930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.915889    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" event={"ID":"3b6f918f-9d67-4933-a823-3d12b2f2da74","Type":"ContainerStarted","Data":"482536f6e9bd3f90be4f669ed952ad371dcf9d76f61c2274f300999c6870003d"}
May 26 11:25:53.916902 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:53.916880    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerStarted","Data":"0bc619175f298d9d08068f98daef306c11032a31a24e003a0609af5afa023ab8"}
May 26 11:25:54.826623 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:54.826585    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:25:54.826892 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:25:54.826647    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:25:54.826892 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:54.826770    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:54.826892 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:54.826812    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:54.826892 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:54.826850    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:10.826828674 +0000 UTC m=+66.787376142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:25:54.826892 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:25:54.826870    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:10.826860028 +0000 UTC m=+66.787407496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:26:01.932669 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:01.932629    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" event={"ID":"3b6f918f-9d67-4933-a823-3d12b2f2da74","Type":"ContainerStarted","Data":"fdd54cf6898a4d9803d1d8735de37adf9f0a465c3dcb1ff5887ca47610d13cf9"}
May 26 11:26:01.933121 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:01.932834    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:26:01.934476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:01.934457    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:26:01.952151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:01.952072    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" podStartSLOduration=1.510132351 podStartE2EDuration="9.952060305s" podCreationTimestamp="2026-05-26 11:25:52 +0000 UTC" firstStartedPulling="2026-05-26 11:25:53.259117093 +0000 UTC m=+49.219664559" lastFinishedPulling="2026-05-26 11:26:01.701045037 +0000 UTC m=+57.661592513" observedRunningTime="2026-05-26 11:26:01.951853406 +0000 UTC m=+57.912400892" watchObservedRunningTime="2026-05-26 11:26:01.952060305 +0000 UTC m=+57.912607797"
May 26 11:26:02.935480 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:02.935439    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerStarted","Data":"95d26070f4222dbf851300ecb0b3396b210bd64e5436a61655a27cf0931ef857"}
May 26 11:26:04.782644 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:04.782615    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8trm"
May 26 11:26:05.943271 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:05.943233    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerStarted","Data":"02fd19bfff62ef5ae3a1ee1b212c031f27e0a7b6fa91fe03a341f004aee41807"}
May 26 11:26:05.943271 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:05.943270    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerStarted","Data":"3e5c7197e0eb7becac3294672f31809836b6c8efaaa36052f335b0c7758e2db0"}
May 26 11:26:05.961259 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:05.961213    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" podStartSLOduration=1.965804618 podStartE2EDuration="13.96119914s" podCreationTimestamp="2026-05-26 11:25:52 +0000 UTC" firstStartedPulling="2026-05-26 11:25:53.295925889 +0000 UTC m=+49.256473352" lastFinishedPulling="2026-05-26 11:26:05.291320408 +0000 UTC m=+61.251867874" observedRunningTime="2026-05-26 11:26:05.960182951 +0000 UTC m=+61.920730448" watchObservedRunningTime="2026-05-26 11:26:05.96119914 +0000 UTC m=+61.921746650"
May 26 11:26:10.342975 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.342936    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:26:10.345679 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.345657    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:26:10.353461 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.353440    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:26:10.353539 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.353503    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:14.353484896 +0000 UTC m=+130.314032364 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : secret "metrics-daemon-secret" not found
May 26 11:26:10.444211 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.444175    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:26:10.447154 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.447137    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:26:10.457013 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.456996    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:26:10.467383 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.467355    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99p8d\" (UniqueName: \"kubernetes.io/projected/659bc118-373e-494f-a4f4-e6deaeea6fed-kube-api-access-99p8d\") pod \"network-check-target-6866p\" (UID: \"659bc118-373e-494f-a4f4-e6deaeea6fed\") " pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:26:10.595121 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.595041    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf2lb\""
May 26 11:26:10.602960 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.602942    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:26:10.727680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.727649    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6866p"]
May 26 11:26:10.730949 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:26:10.730918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659bc118_373e_494f_a4f4_e6deaeea6fed.slice/crio-3e35761497238fcc9d6710be9749a45034d8cbe1f0e5fe066fc7b9bcb5a0e5ac WatchSource:0}: Error finding container 3e35761497238fcc9d6710be9749a45034d8cbe1f0e5fe066fc7b9bcb5a0e5ac: Status 404 returned error can't find the container with id 3e35761497238fcc9d6710be9749a45034d8cbe1f0e5fe066fc7b9bcb5a0e5ac
May 26 11:26:10.847925 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.847831    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:26:10.847925 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.847880    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:26:10.848456 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.847972    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:26:10.848456 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.847995    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:26:10.848456 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.848036    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:42.848010311 +0000 UTC m=+98.808557775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:26:10.848456 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:10.848050    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:42.848043475 +0000 UTC m=+98.808590938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:26:10.955110 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:10.955076    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6866p" event={"ID":"659bc118-373e-494f-a4f4-e6deaeea6fed","Type":"ContainerStarted","Data":"3e35761497238fcc9d6710be9749a45034d8cbe1f0e5fe066fc7b9bcb5a0e5ac"}
May 26 11:26:13.962521 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:13.962488    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6866p" event={"ID":"659bc118-373e-494f-a4f4-e6deaeea6fed","Type":"ContainerStarted","Data":"edfff4f63efbf68c62b2ce8e26d5ba811efb3c4277fa8a185619e0680ad20d7e"}
May 26 11:26:13.962876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:13.962623    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:26:13.983643 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:13.983600    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6866p" podStartSLOduration=67.385074273 podStartE2EDuration="1m9.983590087s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:26:10.732836155 +0000 UTC m=+66.693383623" lastFinishedPulling="2026-05-26 11:26:13.331351971 +0000 UTC m=+69.291899437" observedRunningTime="2026-05-26 11:26:13.982331874 +0000 UTC m=+69.942879360" watchObservedRunningTime="2026-05-26 11:26:13.983590087 +0000 UTC m=+69.944137572"
May 26 11:26:42.872349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:42.872312    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:26:42.872764 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:42.872364    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:26:42.872764 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:42.872469    2562 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:26:42.872764 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:42.872483    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:26:42.872764 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:42.872545    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert podName:817d46b4-0ea1-407a-b72e-7b9292c0a3b8 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:46.872529788 +0000 UTC m=+162.833077251 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert") pod "ingress-canary-x9v7t" (UID: "817d46b4-0ea1-407a-b72e-7b9292c0a3b8") : secret "canary-serving-cert" not found
May 26 11:26:42.872764 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:26:42.872561    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls podName:b140c9d9-dd67-4b37-9745-557d148045d9 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:46.872554276 +0000 UTC m=+162.833101738 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls") pod "dns-default-6ntbs" (UID: "b140c9d9-dd67-4b37-9745-557d148045d9") : secret "dns-default-metrics-tls" not found
May 26 11:26:44.968092 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:26:44.968055    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6866p"
May 26 11:27:14.390095 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:14.390056    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:27:14.390586 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:27:14.390190    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:27:14.390586 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:27:14.390257    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs podName:f8bcaf28-492e-42b7-9252-e3cfa4f217f9 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:16.390238267 +0000 UTC m=+252.350785745 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs") pod "network-metrics-daemon-hkf6m" (UID: "f8bcaf28-492e-42b7-9252-e3cfa4f217f9") : secret "metrics-daemon-secret" not found
May 26 11:27:28.379705 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:28.379672    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lsn6g_c6197213-20ec-4de9-8308-3f48b0cee933/dns-node-resolver/0.log"
May 26 11:27:28.980616 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:28.980591    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8m2x_5cb390ce-0700-4745-9dda-fe590c8284c2/node-ca/0.log"
May 26 11:27:42.006981 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:27:42.006873    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-x9v7t" podUID="817d46b4-0ea1-407a-b72e-7b9292c0a3b8"
May 26 11:27:42.025200 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:27:42.025162    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6ntbs" podUID="b140c9d9-dd67-4b37-9745-557d148045d9"
May 26 11:27:42.166247 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:42.166204    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:27:42.687165 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:27:42.687130    2562 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hkf6m" podUID="f8bcaf28-492e-42b7-9252-e3cfa4f217f9"
May 26 11:27:46.907993 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.907887    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:27:46.907993 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.907956    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:27:46.910342 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.910314    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b140c9d9-dd67-4b37-9745-557d148045d9-metrics-tls\") pod \"dns-default-6ntbs\" (UID: \"b140c9d9-dd67-4b37-9745-557d148045d9\") " pod="openshift-dns/dns-default-6ntbs"
May 26 11:27:46.910454 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.910376    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817d46b4-0ea1-407a-b72e-7b9292c0a3b8-cert\") pod \"ingress-canary-x9v7t\" (UID: \"817d46b4-0ea1-407a-b72e-7b9292c0a3b8\") " pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:27:46.969779 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.969750    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwmqw\""
May 26 11:27:46.977741 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:46.977720    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9v7t"
May 26 11:27:47.096962 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:47.096930    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7t"]
May 26 11:27:47.100904 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:27:47.100878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817d46b4_0ea1_407a_b72e_7b9292c0a3b8.slice/crio-d0c09502571321f12047328a98dcdce7b014c98558699122912d3e8db1de9140 WatchSource:0}: Error finding container d0c09502571321f12047328a98dcdce7b014c98558699122912d3e8db1de9140: Status 404 returned error can't find the container with id d0c09502571321f12047328a98dcdce7b014c98558699122912d3e8db1de9140
May 26 11:27:47.176968 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:47.176854    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9v7t" event={"ID":"817d46b4-0ea1-407a-b72e-7b9292c0a3b8","Type":"ContainerStarted","Data":"d0c09502571321f12047328a98dcdce7b014c98558699122912d3e8db1de9140"}
May 26 11:27:48.082816 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.082782    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m74wn"]
May 26 11:27:48.085777 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.085754    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.095313 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.095293    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:27:48.095598 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.095582    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:27:48.096499 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.096476    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:27:48.096588 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.096485    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lhjtv\""
May 26 11:27:48.096588 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.096487    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:27:48.098190 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.098167    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m74wn"]
May 26 11:27:48.117202 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.117172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzzf\" (UniqueName: \"kubernetes.io/projected/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-api-access-9rzzf\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.117349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.117216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c9d899d-fe8b-4346-bf59-255ebb4f643f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.117349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.117275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9d899d-fe8b-4346-bf59-255ebb4f643f-data-volume\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.117349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.117316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c9d899d-fe8b-4346-bf59-255ebb4f643f-crio-socket\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.117349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.117338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218154 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218105    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c9d899d-fe8b-4346-bf59-255ebb4f643f-crio-socket\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218154 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218155    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218395 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218207    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzzf\" (UniqueName: \"kubernetes.io/projected/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-api-access-9rzzf\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218395 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218243    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c9d899d-fe8b-4346-bf59-255ebb4f643f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218395 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218287    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9d899d-fe8b-4346-bf59-255ebb4f643f-data-volume\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218395 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218246    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c9d899d-fe8b-4346-bf59-255ebb4f643f-crio-socket\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218685 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218664    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9d899d-fe8b-4346-bf59-255ebb4f643f-data-volume\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.218926 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.218882    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.220969 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.220946    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c9d899d-fe8b-4346-bf59-255ebb4f643f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.239326 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.239305    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzzf\" (UniqueName: \"kubernetes.io/projected/0c9d899d-fe8b-4346-bf59-255ebb4f643f-kube-api-access-9rzzf\") pod \"insights-runtime-extractor-m74wn\" (UID: \"0c9d899d-fe8b-4346-bf59-255ebb4f643f\") " pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.395930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.395897    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m74wn"
May 26 11:27:48.670150 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:48.670079    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m74wn"]
May 26 11:27:48.673830 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:27:48.673799    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9d899d_fe8b_4346_bf59_255ebb4f643f.slice/crio-76c58def62f2739fdbcd8ceaf7de29c476fed77d915603a02559b569f71e0353 WatchSource:0}: Error finding container 76c58def62f2739fdbcd8ceaf7de29c476fed77d915603a02559b569f71e0353: Status 404 returned error can't find the container with id 76c58def62f2739fdbcd8ceaf7de29c476fed77d915603a02559b569f71e0353
May 26 11:27:49.182976 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:49.182939    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m74wn" event={"ID":"0c9d899d-fe8b-4346-bf59-255ebb4f643f","Type":"ContainerStarted","Data":"bcc2fc412512e271c90b6a3b67e557f4c42acfee230a916501af0c6ae7be9242"}
May 26 11:27:49.183406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:49.182979    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m74wn" event={"ID":"0c9d899d-fe8b-4346-bf59-255ebb4f643f","Type":"ContainerStarted","Data":"76c58def62f2739fdbcd8ceaf7de29c476fed77d915603a02559b569f71e0353"}
May 26 11:27:49.184336 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:49.184313    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9v7t" event={"ID":"817d46b4-0ea1-407a-b72e-7b9292c0a3b8","Type":"ContainerStarted","Data":"de9bc2dabbf17e4b7ebf62f7d9a2f682d14db6576a70b5c8f441073e7c715e8a"}
May 26 11:27:49.199660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:49.199611    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9v7t" podStartSLOduration=129.713700679 podStartE2EDuration="2m11.199598374s" podCreationTimestamp="2026-05-26 11:25:38 +0000 UTC" firstStartedPulling="2026-05-26 11:27:47.102638642 +0000 UTC m=+163.063186120" lastFinishedPulling="2026-05-26 11:27:48.588536348 +0000 UTC m=+164.549083815" observedRunningTime="2026-05-26 11:27:49.199074369 +0000 UTC m=+165.159621853" watchObservedRunningTime="2026-05-26 11:27:49.199598374 +0000 UTC m=+165.160145859"
May 26 11:27:50.188563 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:50.188529    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m74wn" event={"ID":"0c9d899d-fe8b-4346-bf59-255ebb4f643f","Type":"ContainerStarted","Data":"a070d077e5171795b87ab8d4078f2debf983fa4f59861884ecad19c474c62fe9"}
May 26 11:27:51.193017 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:51.192982    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m74wn" event={"ID":"0c9d899d-fe8b-4346-bf59-255ebb4f643f","Type":"ContainerStarted","Data":"1fb2818eebdeaf16b500bc3b8a92a10ad067329dc8d93f2693741b7af8fc8f06"}
May 26 11:27:51.214727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:51.214684    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m74wn" podStartSLOduration=1.47628398 podStartE2EDuration="3.214671674s" podCreationTimestamp="2026-05-26 11:27:48 +0000 UTC" firstStartedPulling="2026-05-26 11:27:48.734067614 +0000 UTC m=+164.694615077" lastFinishedPulling="2026-05-26 11:27:50.472455305 +0000 UTC m=+166.433002771" observedRunningTime="2026-05-26 11:27:51.214474792 +0000 UTC m=+167.175022277" watchObservedRunningTime="2026-05-26 11:27:51.214671674 +0000 UTC m=+167.175219195"
May 26 11:27:56.673990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:56.673958    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ntbs"
May 26 11:27:56.676907 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:56.676888    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6qhf\""
May 26 11:27:56.684856 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:56.684840    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ntbs"
May 26 11:27:56.809548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:56.808478    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ntbs"]
May 26 11:27:56.813737 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:27:56.813708    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb140c9d9_dd67_4b37_9745_557d148045d9.slice/crio-8684e3c2df832f8e5e3f07896a079f6f3d40d80764a13c3ffeebb0b2beade12b WatchSource:0}: Error finding container 8684e3c2df832f8e5e3f07896a079f6f3d40d80764a13c3ffeebb0b2beade12b: Status 404 returned error can't find the container with id 8684e3c2df832f8e5e3f07896a079f6f3d40d80764a13c3ffeebb0b2beade12b
May 26 11:27:57.206965 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:57.206915    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ntbs" event={"ID":"b140c9d9-dd67-4b37-9745-557d148045d9","Type":"ContainerStarted","Data":"8684e3c2df832f8e5e3f07896a079f6f3d40d80764a13c3ffeebb0b2beade12b"}
May 26 11:27:57.674150 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:57.674118    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:27:58.211264 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:58.211234    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ntbs" event={"ID":"b140c9d9-dd67-4b37-9745-557d148045d9","Type":"ContainerStarted","Data":"bac90d511566bfe40fe51195b6e2588aaa97b67ae4fff7d6e17ca21800d198d7"}
May 26 11:27:58.211380 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:58.211272    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ntbs" event={"ID":"b140c9d9-dd67-4b37-9745-557d148045d9","Type":"ContainerStarted","Data":"c42c3db386d724115f0e9ec9c2e67bb74334a3663634d9d592d44257fcaa3ae3"}
May 26 11:27:58.211380 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:58.211379    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6ntbs"
May 26 11:27:58.231931 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:27:58.231885    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-6ntbs" podStartSLOduration=139.012030283 podStartE2EDuration="2m20.231870536s" podCreationTimestamp="2026-05-26 11:25:38 +0000 UTC" firstStartedPulling="2026-05-26 11:27:56.815353061 +0000 UTC m=+172.775900527" lastFinishedPulling="2026-05-26 11:27:58.035193317 +0000 UTC m=+173.995740780" observedRunningTime="2026-05-26 11:27:58.230409082 +0000 UTC m=+174.190956562" watchObservedRunningTime="2026-05-26 11:27:58.231870536 +0000 UTC m=+174.192418020"
May 26 11:28:01.933606 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:01.933560    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" podUID="3b6f918f-9d67-4933-a823-3d12b2f2da74" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.7:8000/readyz\": dial tcp 10.134.0.7:8000: connect: connection refused"
May 26 11:28:02.220132 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:02.220098    2562 generic.go:358] "Generic (PLEG): container finished" podID="3b6f918f-9d67-4933-a823-3d12b2f2da74" containerID="fdd54cf6898a4d9803d1d8735de37adf9f0a465c3dcb1ff5887ca47610d13cf9" exitCode=1
May 26 11:28:02.220266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:02.220174    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" event={"ID":"3b6f918f-9d67-4933-a823-3d12b2f2da74","Type":"ContainerDied","Data":"fdd54cf6898a4d9803d1d8735de37adf9f0a465c3dcb1ff5887ca47610d13cf9"}
May 26 11:28:02.220523 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:02.220507    2562 scope.go:117] "RemoveContainer" containerID="fdd54cf6898a4d9803d1d8735de37adf9f0a465c3dcb1ff5887ca47610d13cf9"
May 26 11:28:03.091460 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:03.091398    2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:28:03.223670 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:03.223638    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx" event={"ID":"3b6f918f-9d67-4933-a823-3d12b2f2da74","Type":"ContainerStarted","Data":"443f96627eef7e9188d8cae1be413c8b5c90564221eaa040fb3841e580c8c19c"}
May 26 11:28:03.223829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:03.223819    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:28:03.224486 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:03.224458    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8c7ff64f6-vzrsx"
May 26 11:28:04.202638 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.202602    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p8k6h"]
May 26 11:28:04.207301 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.207278    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.211850 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.211822    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:28:04.212102 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.212075    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:28:04.212182 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.212076    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5tkgs\""
May 26 11:28:04.213203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.213177    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:28:04.213203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.213194    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:28:04.213468 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.213450    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:28:04.217785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.217767    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:28:04.331454 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-wtmp\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-root\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmh66\" (UniqueName: \"kubernetes.io/projected/1cb83698-8cf8-40ad-b882-54745cdf375f-kube-api-access-wmh66\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-sys\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-textfile\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-metrics-client-ca\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.331805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.331771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.432890 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.432853    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.432890 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.432893    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.432915    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-wtmp\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433038    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-wtmp\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433035    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-root\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433069    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-root\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433073    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmh66\" (UniqueName: \"kubernetes.io/projected/1cb83698-8cf8-40ad-b882-54745cdf375f-kube-api-access-wmh66\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433143    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-sys\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433164    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433183    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-textfile\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433201    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-metrics-client-ca\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433251    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cb83698-8cf8-40ad-b882-54745cdf375f-sys\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:28:04.433287    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 11:28:04.433376 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:28:04.433357    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls podName:1cb83698-8cf8-40ad-b882-54745cdf375f nodeName:}" failed. No retries permitted until 2026-05-26 11:28:04.933336433 +0000 UTC m=+180.893883905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls") pod "node-exporter-p8k6h" (UID: "1cb83698-8cf8-40ad-b882-54745cdf375f") : secret "node-exporter-tls" not found
May 26 11:28:04.433720 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433514    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-textfile\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433720 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433598    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.433786 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.433731    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cb83698-8cf8-40ad-b882-54745cdf375f-metrics-client-ca\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.435180 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.435157    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.442031 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.442014    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmh66\" (UniqueName: \"kubernetes.io/projected/1cb83698-8cf8-40ad-b882-54745cdf375f-kube-api-access-wmh66\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.937900 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.937861    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:04.940125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:04.940106    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1cb83698-8cf8-40ad-b882-54745cdf375f-node-exporter-tls\") pod \"node-exporter-p8k6h\" (UID: \"1cb83698-8cf8-40ad-b882-54745cdf375f\") " pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:05.119100 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:05.119071    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5tkgs\""
May 26 11:28:05.127498 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:05.127479    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8k6h"
May 26 11:28:05.135139 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:28:05.135115    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb83698_8cf8_40ad_b882_54745cdf375f.slice/crio-f152103b33f889890e0c2fc52e4168640282cdbbfc731d353995375bf33b29d8 WatchSource:0}: Error finding container f152103b33f889890e0c2fc52e4168640282cdbbfc731d353995375bf33b29d8: Status 404 returned error can't find the container with id f152103b33f889890e0c2fc52e4168640282cdbbfc731d353995375bf33b29d8
May 26 11:28:05.230126 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:05.230041    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8k6h" event={"ID":"1cb83698-8cf8-40ad-b882-54745cdf375f","Type":"ContainerStarted","Data":"f152103b33f889890e0c2fc52e4168640282cdbbfc731d353995375bf33b29d8"}
May 26 11:28:06.233607 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.233514    2562 generic.go:358] "Generic (PLEG): container finished" podID="1cb83698-8cf8-40ad-b882-54745cdf375f" containerID="3dc53654a07fe81bab0b2a467d618e6407e642079d9a4fb145a6e3b382be2a00" exitCode=0
May 26 11:28:06.233607 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.233559    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8k6h" event={"ID":"1cb83698-8cf8-40ad-b882-54745cdf375f","Type":"ContainerDied","Data":"3dc53654a07fe81bab0b2a467d618e6407e642079d9a4fb145a6e3b382be2a00"}
May 26 11:28:06.284914 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.284890    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56588f9995-tb5j5"]
May 26 11:28:06.290068 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.290042    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.293697 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.293673    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 26 11:28:06.293697 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.293684    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 26 11:28:06.293825 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.293706    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 26 11:28:06.294194 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.294176    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 26 11:28:06.294379 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.294361    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-frwcx\""
May 26 11:28:06.294794 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.294775    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2r24a7s134l5p\""
May 26 11:28:06.294879 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.294843    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 26 11:28:06.316990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.316964    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56588f9995-tb5j5"]
May 26 11:28:06.347255 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347332 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347447 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a60299-e747-474a-b063-70c60118e2b8-metrics-client-ca\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347535 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-grpc-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347535 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbs8n\" (UniqueName: \"kubernetes.io/projected/67a60299-e747-474a-b063-70c60118e2b8-kube-api-access-zbs8n\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347612 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347612 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.347713 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.347681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448644 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448603    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448655    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448693    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448729    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a60299-e747-474a-b063-70c60118e2b8-metrics-client-ca\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448762    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-grpc-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.448818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448786    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbs8n\" (UniqueName: \"kubernetes.io/projected/67a60299-e747-474a-b063-70c60118e2b8-kube-api-access-zbs8n\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.449047 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448837    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.449047 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.448865    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.449889 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.449857    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a60299-e747-474a-b063-70c60118e2b8-metrics-client-ca\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.451345 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.451321    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.451509 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.451488    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.451555 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.451518    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.451658 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.451636    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-grpc-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.452071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.452052    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.452181 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.452163    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67a60299-e747-474a-b063-70c60118e2b8-secret-thanos-querier-tls\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.456582 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.456559    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbs8n\" (UniqueName: \"kubernetes.io/projected/67a60299-e747-474a-b063-70c60118e2b8-kube-api-access-zbs8n\") pod \"thanos-querier-56588f9995-tb5j5\" (UID: \"67a60299-e747-474a-b063-70c60118e2b8\") " pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.600597 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.600507    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:06.732887 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:06.732845    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56588f9995-tb5j5"]
May 26 11:28:06.735337 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:28:06.735302    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a60299_e747_474a_b063_70c60118e2b8.slice/crio-faa90176809b1bfd6874739a1842dc50ec321b2a5d3b8cfe0ae56fa90e4b49f4 WatchSource:0}: Error finding container faa90176809b1bfd6874739a1842dc50ec321b2a5d3b8cfe0ae56fa90e4b49f4: Status 404 returned error can't find the container with id faa90176809b1bfd6874739a1842dc50ec321b2a5d3b8cfe0ae56fa90e4b49f4
May 26 11:28:07.237278 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:07.237213    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"faa90176809b1bfd6874739a1842dc50ec321b2a5d3b8cfe0ae56fa90e4b49f4"}
May 26 11:28:07.239476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:07.239446    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8k6h" event={"ID":"1cb83698-8cf8-40ad-b882-54745cdf375f","Type":"ContainerStarted","Data":"a8a6250887900282d92849b07be95310a6f54064156e1a23e1cf0851b5f4ff75"}
May 26 11:28:07.239630 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:07.239483    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8k6h" event={"ID":"1cb83698-8cf8-40ad-b882-54745cdf375f","Type":"ContainerStarted","Data":"da5a0a5fbd8fd08700accc1d13599c2e6129686e5fe32ec79c5b2a2453102749"}
May 26 11:28:07.261521 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:07.261451    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p8k6h" podStartSLOduration=2.447552653 podStartE2EDuration="3.261436984s" podCreationTimestamp="2026-05-26 11:28:04 +0000 UTC" firstStartedPulling="2026-05-26 11:28:05.136983792 +0000 UTC m=+181.097531254" lastFinishedPulling="2026-05-26 11:28:05.950868119 +0000 UTC m=+181.911415585" observedRunningTime="2026-05-26 11:28:07.259826059 +0000 UTC m=+183.220373545" watchObservedRunningTime="2026-05-26 11:28:07.261436984 +0000 UTC m=+183.221984471"
May 26 11:28:08.216653 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.216615    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6ntbs"
May 26 11:28:08.680582 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.680548    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fcc9475df-q4q8c"]
May 26 11:28:08.683464 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.683446    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.686135 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.686116    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 26 11:28:08.686793 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.686779    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 11:28:08.686885 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.686810    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bs6tjle7okjih\""
May 26 11:28:08.687103 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.687083    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 26 11:28:08.687175 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.687091    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m7wdz\""
May 26 11:28:08.687175 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.687127    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 26 11:28:08.695495 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.695476    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fcc9475df-q4q8c"]
May 26 11:28:08.767755 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.767724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-tls\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.767877 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.767775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.767877 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.767847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-metrics-server-audit-profiles\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.767999 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.767902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wlh\" (UniqueName: \"kubernetes.io/projected/50c18f3f-f750-40ef-b5cf-db3beb602df5-kube-api-access-86wlh\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.767999 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.767957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-client-certs\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.768095 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.768000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-client-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.768095 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.768026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50c18f3f-f750-40ef-b5cf-db3beb602df5-audit-log\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869275 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869222    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-tls\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869469 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869378    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869553 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869430    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-metrics-server-audit-profiles\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869551    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86wlh\" (UniqueName: \"kubernetes.io/projected/50c18f3f-f750-40ef-b5cf-db3beb602df5-kube-api-access-86wlh\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869652 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869622    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-client-certs\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869673    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-client-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.869751 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.869698    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50c18f3f-f750-40ef-b5cf-db3beb602df5-audit-log\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.870180 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.870152    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50c18f3f-f750-40ef-b5cf-db3beb602df5-audit-log\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.870301 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.870201    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.870535 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.870504    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50c18f3f-f750-40ef-b5cf-db3beb602df5-metrics-server-audit-profiles\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.872385 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.872361    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-client-ca-bundle\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.872507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.872481    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-client-certs\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.872576 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.872500    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50c18f3f-f750-40ef-b5cf-db3beb602df5-secret-metrics-server-tls\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.877173 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.877152    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wlh\" (UniqueName: \"kubernetes.io/projected/50c18f3f-f750-40ef-b5cf-db3beb602df5-kube-api-access-86wlh\") pod \"metrics-server-5fcc9475df-q4q8c\" (UID: \"50c18f3f-f750-40ef-b5cf-db3beb602df5\") " pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:08.929977 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.929940    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"]
May 26 11:28:08.936120 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.936053    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:08.938509 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.938484    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 26 11:28:08.938660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.938641    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-9krfs\""
May 26 11:28:08.941895 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.941869    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"]
May 26 11:28:08.991977 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:08.991939    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:09.071065 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.071041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5ljjc\" (UID: \"9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:09.165448 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.165408    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fcc9475df-q4q8c"]
May 26 11:28:09.168774 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:28:09.168752    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c18f3f_f750_40ef_b5cf_db3beb602df5.slice/crio-d3068cd08878f7a3bd52107495281809eb8402ecdf22921aa3ff2954afd830a1 WatchSource:0}: Error finding container d3068cd08878f7a3bd52107495281809eb8402ecdf22921aa3ff2954afd830a1: Status 404 returned error can't find the container with id d3068cd08878f7a3bd52107495281809eb8402ecdf22921aa3ff2954afd830a1
May 26 11:28:09.171737 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.171717    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5ljjc\" (UID: \"9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:09.173972 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.173905    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5ljjc\" (UID: \"9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:09.247066 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.247034    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:09.248213 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.248187    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c" event={"ID":"50c18f3f-f750-40ef-b5cf-db3beb602df5","Type":"ContainerStarted","Data":"d3068cd08878f7a3bd52107495281809eb8402ecdf22921aa3ff2954afd830a1"}
May 26 11:28:09.251128 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251097    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"bec2768c11c602fa863504dad65d54807d48ef4a2e7d5d4cfa4cd04f8a8ba78b"}
May 26 11:28:09.251218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251134    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"78266b6c241171c626a5fb3c20f2fc1a49b6fd1489079b721bb80af20eb84c6e"}
May 26 11:28:09.251218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251148    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"5b47c1f2a62e61470eca7d276590d81dd418c45a7835c8a9614d1b5339990982"}
May 26 11:28:09.251218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251161    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"e2d1f530e070d58c2617f15adf87d6530443b392362eaf9eca9e2c345a87a3f2"}
May 26 11:28:09.251218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251173    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"77926b8d700da630cc0b4cfb9a9dca4d87e92ceb7efab1fe523a340dfa5770ce"}
May 26 11:28:09.251218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251188    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" event={"ID":"67a60299-e747-474a-b063-70c60118e2b8","Type":"ContainerStarted","Data":"19af9fa72f2cb53b8294d17bb8a103a2b7e0bfb61d8f3856b9afcfb49cbfc91c"}
May 26 11:28:09.251544 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.251270    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:09.276466 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.276397    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5" podStartSLOduration=0.958837336 podStartE2EDuration="3.276381598s" podCreationTimestamp="2026-05-26 11:28:06 +0000 UTC" firstStartedPulling="2026-05-26 11:28:06.737225861 +0000 UTC m=+182.697773325" lastFinishedPulling="2026-05-26 11:28:09.054770118 +0000 UTC m=+185.015317587" observedRunningTime="2026-05-26 11:28:09.274879995 +0000 UTC m=+185.235427480" watchObservedRunningTime="2026-05-26 11:28:09.276381598 +0000 UTC m=+185.236929083"
May 26 11:28:09.388723 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:09.388694    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"]
May 26 11:28:09.391851 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:28:09.391826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6deeb7_8f24_445d_bfc8_a5a1fbd4c245.slice/crio-faa7aabcb2f695956a279919db8941b68f802f16333c22f98f07d9950490efad WatchSource:0}: Error finding container faa7aabcb2f695956a279919db8941b68f802f16333c22f98f07d9950490efad: Status 404 returned error can't find the container with id faa7aabcb2f695956a279919db8941b68f802f16333c22f98f07d9950490efad
May 26 11:28:10.255391 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.255352    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc" event={"ID":"9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245","Type":"ContainerStarted","Data":"faa7aabcb2f695956a279919db8941b68f802f16333c22f98f07d9950490efad"}
May 26 11:28:10.550615 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.550533    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:28:10.554486 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.554462    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.557712 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.557677    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 11:28:10.558589 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558570    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 11:28:10.558688 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558641    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 11:28:10.558917 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558814    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 11:28:10.558917 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558842    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 11:28:10.558917 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558821    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 11:28:10.559120 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.558889    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5182tc4kml6r6\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559555    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559574    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559574    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559606    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559618    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8bv2p\""
May 26 11:28:10.559655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.559555    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 11:28:10.561298 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.561280    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 11:28:10.604554 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.604524    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:28:10.684809 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.684988 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.684975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbsl7\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.685434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.685251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.785948 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.785909    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786109 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.785957    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786109 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.785990    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786109 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786011    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786265 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786122    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786265 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786168    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pbsl7\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786265 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786200    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786333    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786394    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786534 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786441    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786534 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786490    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786534 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786519    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786550    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786589    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786622    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786651    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786676    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786909 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786729    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786909 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786796    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.786909 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.786802    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.788949    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.789124    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.789445    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.789806    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.789821    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.789957    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.790190    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.790514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.790320    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.791010 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.790526    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.791679 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.791650    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.792136 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.792115    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.792324 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.792308    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.792619 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.792599    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.793031 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.792836    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.793264 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.793248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.808047 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.808022    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbsl7\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7\") pod \"prometheus-k8s-0\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:10.866504 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:10.866479    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:11.013124 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.012903    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:28:11.016938 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:28:11.016914    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8da6e5e_1a4e_43fa_ae8a_39071eb4eb83.slice/crio-d777be601e2450332870f8317073ea193ab9b113f1cf180f4996f4740add1dad WatchSource:0}: Error finding container d777be601e2450332870f8317073ea193ab9b113f1cf180f4996f4740add1dad: Status 404 returned error can't find the container with id d777be601e2450332870f8317073ea193ab9b113f1cf180f4996f4740add1dad
May 26 11:28:11.259429 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.259376    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc" event={"ID":"9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245","Type":"ContainerStarted","Data":"523c9aa60efd520f06185ee6e70fecaec5198ae21494f92247ff4ce980d91874"}
May 26 11:28:11.259861 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.259616    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:11.260628 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.260597    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"d777be601e2450332870f8317073ea193ab9b113f1cf180f4996f4740add1dad"}
May 26 11:28:11.261914 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.261887    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c" event={"ID":"50c18f3f-f750-40ef-b5cf-db3beb602df5","Type":"ContainerStarted","Data":"587c8594e7b0215b0a488a5e840b7b27018ce10b9da0b72c2791911d3ed55d95"}
May 26 11:28:11.264684 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.264664    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc"
May 26 11:28:11.281408 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.281368    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5ljjc" podStartSLOduration=1.874966132 podStartE2EDuration="3.281356169s" podCreationTimestamp="2026-05-26 11:28:08 +0000 UTC" firstStartedPulling="2026-05-26 11:28:09.394143355 +0000 UTC m=+185.354690832" lastFinishedPulling="2026-05-26 11:28:10.800533404 +0000 UTC m=+186.761080869" observedRunningTime="2026-05-26 11:28:11.280793653 +0000 UTC m=+187.241341139" watchObservedRunningTime="2026-05-26 11:28:11.281356169 +0000 UTC m=+187.241903653"
May 26 11:28:11.302764 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:11.302729    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c" podStartSLOduration=1.677654188 podStartE2EDuration="3.302716801s" podCreationTimestamp="2026-05-26 11:28:08 +0000 UTC" firstStartedPulling="2026-05-26 11:28:09.170806855 +0000 UTC m=+185.131354320" lastFinishedPulling="2026-05-26 11:28:10.79586947 +0000 UTC m=+186.756416933" observedRunningTime="2026-05-26 11:28:11.301860926 +0000 UTC m=+187.262408415" watchObservedRunningTime="2026-05-26 11:28:11.302716801 +0000 UTC m=+187.263264285"
May 26 11:28:12.266549 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:12.266460    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98" exitCode=0
May 26 11:28:12.266992 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:12.266565    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"}
May 26 11:28:15.261448 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.261349    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56588f9995-tb5j5"
May 26 11:28:15.277098 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277065    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"}
May 26 11:28:15.277204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277103    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"}
May 26 11:28:15.277204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277117    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"}
May 26 11:28:15.277204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277130    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"}
May 26 11:28:15.277204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277143    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"}
May 26 11:28:15.277204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.277155    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerStarted","Data":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"}
May 26 11:28:15.327710 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.327660    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.701104043 podStartE2EDuration="5.327643239s" podCreationTimestamp="2026-05-26 11:28:10 +0000 UTC" firstStartedPulling="2026-05-26 11:28:11.018999606 +0000 UTC m=+186.979547070" lastFinishedPulling="2026-05-26 11:28:14.645538803 +0000 UTC m=+190.606086266" observedRunningTime="2026-05-26 11:28:15.325730152 +0000 UTC m=+191.286277700" watchObservedRunningTime="2026-05-26 11:28:15.327643239 +0000 UTC m=+191.288190755"
May 26 11:28:15.867553 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:15.867510    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:28:23.154014 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:23.153972    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" podUID="9219ee55-493b-4526-acec-a9c39193d91f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:28:28.993064 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:28.993025    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:28.993064 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:28.993073    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:33.153765 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:33.153725    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" podUID="9219ee55-493b-4526-acec-a9c39193d91f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:28:37.404763 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:37.404737    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ntbs_b140c9d9-dd67-4b37-9745-557d148045d9/dns/0.log"
May 26 11:28:37.415365 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:37.415337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ntbs_b140c9d9-dd67-4b37-9745-557d148045d9/kube-rbac-proxy/0.log"
May 26 11:28:37.971339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:37.971313    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lsn6g_c6197213-20ec-4de9-8308-3f48b0cee933/dns-node-resolver/0.log"
May 26 11:28:43.154236 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.154194    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" podUID="9219ee55-493b-4526-acec-a9c39193d91f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 11:28:43.154616 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.154273    2562 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj"
May 26 11:28:43.154785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.154756    2562 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"02fd19bfff62ef5ae3a1ee1b212c031f27e0a7b6fa91fe03a341f004aee41807"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 26 11:28:43.154831 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.154816    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" podUID="9219ee55-493b-4526-acec-a9c39193d91f" containerName="service-proxy" containerID="cri-o://02fd19bfff62ef5ae3a1ee1b212c031f27e0a7b6fa91fe03a341f004aee41807" gracePeriod=30
May 26 11:28:43.352930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.352894    2562 generic.go:358] "Generic (PLEG): container finished" podID="9219ee55-493b-4526-acec-a9c39193d91f" containerID="02fd19bfff62ef5ae3a1ee1b212c031f27e0a7b6fa91fe03a341f004aee41807" exitCode=2
May 26 11:28:43.353091 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.352939    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerDied","Data":"02fd19bfff62ef5ae3a1ee1b212c031f27e0a7b6fa91fe03a341f004aee41807"}
May 26 11:28:43.353091 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:43.352965    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4788496-qfhzj" event={"ID":"9219ee55-493b-4526-acec-a9c39193d91f","Type":"ContainerStarted","Data":"7c6f262daa5ec7ebae3fe6f8adc197ea2b685077b95470bc4065779bb9824b9e"}
May 26 11:28:49.004000 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:49.003959    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:28:49.008244 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:28:49.008216    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fcc9475df-q4q8c"
May 26 11:29:10.867204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:10.867169    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:10.884111 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:10.884087    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:11.440559 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:11.440533    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:16.440693 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:16.440613    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:29:16.442751 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:16.442719    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8bcaf28-492e-42b7-9252-e3cfa4f217f9-metrics-certs\") pod \"network-metrics-daemon-hkf6m\" (UID: \"f8bcaf28-492e-42b7-9252-e3cfa4f217f9\") " pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:29:16.580568 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:16.580536    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gx5q\""
May 26 11:29:16.585953 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:16.585936    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkf6m"
May 26 11:29:16.728511 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:16.728399    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hkf6m"]
May 26 11:29:16.731429 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:29:16.731383    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bcaf28_492e_42b7_9252_e3cfa4f217f9.slice/crio-6794b3ba56f7cb87231b9da0a62e516e676490a6a314970e06f241af62e4137b WatchSource:0}: Error finding container 6794b3ba56f7cb87231b9da0a62e516e676490a6a314970e06f241af62e4137b: Status 404 returned error can't find the container with id 6794b3ba56f7cb87231b9da0a62e516e676490a6a314970e06f241af62e4137b
May 26 11:29:17.443990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:17.443944    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkf6m" event={"ID":"f8bcaf28-492e-42b7-9252-e3cfa4f217f9","Type":"ContainerStarted","Data":"6794b3ba56f7cb87231b9da0a62e516e676490a6a314970e06f241af62e4137b"}
May 26 11:29:18.448217 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:18.448183    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkf6m" event={"ID":"f8bcaf28-492e-42b7-9252-e3cfa4f217f9","Type":"ContainerStarted","Data":"24bab78a2483990e6d24f9ab315bd56b022a2a10a1ce77ede29de6204ee883dc"}
May 26 11:29:18.448689 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:18.448226    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkf6m" event={"ID":"f8bcaf28-492e-42b7-9252-e3cfa4f217f9","Type":"ContainerStarted","Data":"2d75923283e2ec6ef63fa89cfa385d808a89baa28d7eb33ae4f9b401723cb334"}
May 26 11:29:18.465521 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:18.465473    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hkf6m" podStartSLOduration=253.589802426 podStartE2EDuration="4m14.465460517s" podCreationTimestamp="2026-05-26 11:25:04 +0000 UTC" firstStartedPulling="2026-05-26 11:29:16.733159514 +0000 UTC m=+252.693706981" lastFinishedPulling="2026-05-26 11:29:17.608817606 +0000 UTC m=+253.569365072" observedRunningTime="2026-05-26 11:29:18.464283193 +0000 UTC m=+254.424830690" watchObservedRunningTime="2026-05-26 11:29:18.465460517 +0000 UTC m=+254.426008043"
May 26 11:29:28.939104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939066    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939507    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="prometheus" containerID="cri-o://41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126" gracePeriod=600
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939552    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="thanos-sidecar" containerID="cri-o://08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38" gracePeriod=600
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939580    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2" gracePeriod=600
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939586    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="config-reloader" containerID="cri-o://fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d" gracePeriod=600
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939531    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy" containerID="cri-o://1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9" gracePeriod=600
May 26 11:29:28.939682 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:28.939581    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-web" containerID="cri-o://c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0" gracePeriod=600
May 26 11:29:29.178911 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.178888    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.341062 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.340969    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341062 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341010    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341062 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341027    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341062 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341051    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341076    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341104    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341135    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341170    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341651 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341621    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:29.341716 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341699    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341770 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341739    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341821 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341776    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341821 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341801    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341923 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341836    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341923 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341881    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbsl7\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.341923 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341913    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.342070 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341950    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.342070 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.341982    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.342070 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.342039    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config\") pod \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\" (UID: \"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83\") "
May 26 11:29:29.342295 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.342277    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.342680 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.342647    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:29.343550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.343521    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:29.343728 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.343698    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.344055 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344018    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.344504 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344476    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.344602 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344581    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.344801 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344754    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config" (OuterVolumeSpecName: "config") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.345269 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344935    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:29.345269 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.344992    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.345269 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.345197    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:29.345269 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.345221    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:29:29.345728 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.345704    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:29:29.345906 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.345869    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.346607 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.346584    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.346976 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.346954    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out" (OuterVolumeSpecName: "config-out") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:29:29.347055 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.346983    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7" (OuterVolumeSpecName: "kube-api-access-pbsl7") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "kube-api-access-pbsl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:29:29.353973 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.353943    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config" (OuterVolumeSpecName: "web-config") pod "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" (UID: "b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:29.443104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443070    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-grpc-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443099    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443110    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-kube-rbac-proxy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443120    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-metrics-client-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443129    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-web-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443138    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443147    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443156    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-tls-assets\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443164    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443174    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-metrics-client-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443182    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config-out\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443190    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443199    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbsl7\" (UniqueName: \"kubernetes.io/projected/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-kube-api-access-pbsl7\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443207    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443216    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-prometheus-k8s-db\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443224    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.443311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.443233    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:29:29.480387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480359    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2" exitCode=0
May 26 11:29:29.480387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480382    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9" exitCode=0
May 26 11:29:29.480387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480389    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0" exitCode=0
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480397    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38" exitCode=0
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480403    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d" exitCode=0
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480408    2562 generic.go:358] "Generic (PLEG): container finished" podID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126" exitCode=0
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480462    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"}
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480515    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480530    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.480642 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480515    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"}
May 26 11:29:29.480886 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480645    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"}
May 26 11:29:29.480886 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480661    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"}
May 26 11:29:29.480886 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480673    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"}
May 26 11:29:29.480886 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480686    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"}
May 26 11:29:29.480886 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.480698    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83","Type":"ContainerDied","Data":"d777be601e2450332870f8317073ea193ab9b113f1cf180f4996f4740add1dad"}
May 26 11:29:29.488059 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.488000    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.495069 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.495051    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.501274 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.501257    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.504361 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.504338    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:29.508469 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.508409    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.509563 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.509542    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:29.514681 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.514663    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.520952 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.520937    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.526782 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.526768    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.527003 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.526985    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.527069 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527014    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.527069 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527041    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.527273 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.527257    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.527311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527279    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.527311 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527303    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.527516 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.527500    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.527582 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527524    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.527582 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527543    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.527755 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.527735    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.527792 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527760    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.527792 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527773    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.527960 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.527945    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.528018 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527968    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.528018 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.527986    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.528224 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.528208    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.528272 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528229    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.528272 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528244    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.528494 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:29:29.528453    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.528494 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528471    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.528494 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528482    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.528657 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528642    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.528657 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528656    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.528885 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528866    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.528971 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.528886    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.529151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529130    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.529218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529153    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.529354 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529339    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.529434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529355    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.529587 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529570    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.529653 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529587    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.529787 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529772    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.529850 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529788    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.529992 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529974    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.530056 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.529995    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.530227 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530209    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.530271 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530229    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.530462 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.530551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530463    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.530650 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530633    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.530691 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530650    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.530823 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530809    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.530869 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530824    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.530986 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530967    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.531039 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.530987    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.531165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531151    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.531203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531165    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.531335 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531316    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.531399 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531336    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.531552 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531535    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.531607 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531554    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.531731 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531713    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.531793 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531732    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.531930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531915    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.531975 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.531930    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.532104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532090    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.532151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532104    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.532266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532253    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.532316 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532266    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.532456 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532439    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.532456 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532456    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.532631 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532616    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.532677 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532631    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.532790 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532777    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.532832 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532790    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.532973 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532959    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.533022 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.532973    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.533143 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533128    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.533188 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533144    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.533319 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533306    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.533364 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533320    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.533507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533489    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.533581 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533510    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.533699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533684    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.533699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533698    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.533898 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533882    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.533961 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.533899    2562 scope.go:117] "RemoveContainer" containerID="e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"
May 26 11:29:29.534105 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534088    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2"} err="failed to get container status \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": rpc error: code = NotFound desc = could not find container \"e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2\": container with ID starting with e07085da0e718ee3f0a74c4a1552aedfa78a493f15b49f01c6595d88ef0968d2 not found: ID does not exist"
May 26 11:29:29.534179 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534105    2562 scope.go:117] "RemoveContainer" containerID="1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"
May 26 11:29:29.534309 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534290    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9"} err="failed to get container status \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": rpc error: code = NotFound desc = could not find container \"1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9\": container with ID starting with 1e6953b3d3d12df311cd68e5a20f9fd33b8c46188ef700e64657c92e0c101ec9 not found: ID does not exist"
May 26 11:29:29.534360 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534310    2562 scope.go:117] "RemoveContainer" containerID="c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"
May 26 11:29:29.534534 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534518    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0"} err="failed to get container status \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": rpc error: code = NotFound desc = could not find container \"c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0\": container with ID starting with c041463b83a8ac5ad74b57c3c1760a5f5151464ae191ecad04d5ad4836fe92a0 not found: ID does not exist"
May 26 11:29:29.534589 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534535    2562 scope.go:117] "RemoveContainer" containerID="08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"
May 26 11:29:29.534726 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534707    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38"} err="failed to get container status \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": rpc error: code = NotFound desc = could not find container \"08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38\": container with ID starting with 08296dfb0bdbc77103b679edd19c4ef3011ff09c16d84a479acda220376b3c38 not found: ID does not exist"
May 26 11:29:29.534776 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534725    2562 scope.go:117] "RemoveContainer" containerID="fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"
May 26 11:29:29.534903 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534888    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d"} err="failed to get container status \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": rpc error: code = NotFound desc = could not find container \"fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d\": container with ID starting with fa392ba7fac2e0eec400809021aed0c71dd7267f7d37dfaae85d40d28439560d not found: ID does not exist"
May 26 11:29:29.534903 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.534902    2562 scope.go:117] "RemoveContainer" containerID="41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"
May 26 11:29:29.535078 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.535062    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126"} err="failed to get container status \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": rpc error: code = NotFound desc = could not find container \"41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126\": container with ID starting with 41f0adf7046ff4184ebd9f0ccd04205ccdb68adb62cf427d972264d9f6523126 not found: ID does not exist"
May 26 11:29:29.535078 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.535076    2562 scope.go:117] "RemoveContainer" containerID="76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"
May 26 11:29:29.535257 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.535240    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98"} err="failed to get container status \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": rpc error: code = NotFound desc = could not find container \"76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98\": container with ID starting with 76795631c5b659e2e87861274ce2af6dc522aa334855a892baa0ea006d8c7e98 not found: ID does not exist"
May 26 11:29:29.539747 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.539726    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:29.540017 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540002    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-thanos"
May 26 11:29:29.540084 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540020    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-thanos"
May 26 11:29:29.540084 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540046    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy"
May 26 11:29:29.540084 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540056    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy"
May 26 11:29:29.540084 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540072    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="config-reloader"
May 26 11:29:29.540084 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540081    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="config-reloader"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540095    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="init-config-reloader"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540104    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="init-config-reloader"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540115    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="thanos-sidecar"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540123    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="thanos-sidecar"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540135    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="prometheus"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540143    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="prometheus"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-web"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540160    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-web"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540222    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="config-reloader"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540234    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540246    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="prometheus"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540256    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="thanos-sidecar"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540265    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-web"
May 26 11:29:29.540317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.540275    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" containerName="kube-rbac-proxy-thanos"
May 26 11:29:29.545490 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.545472    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.548101 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548079    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 11:29:29.548196 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548079    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5182tc4kml6r6\""
May 26 11:29:29.548196 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548150    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 11:29:29.548570 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548550    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 11:29:29.548660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548588    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 11:29:29.548660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548588    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 11:29:29.548937 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548921    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 11:29:29.549067 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.548927    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 11:29:29.549194 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.549177    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 11:29:29.549502 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.549473    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 11:29:29.549675 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.549505    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 11:29:29.549675 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.549516    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8bv2p\""
May 26 11:29:29.552740 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.552718    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 11:29:29.554406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.554388    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 11:29:29.558754 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.558729    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:29.644957 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.644919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.644965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.644986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645125 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcbm\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-kube-api-access-fbcbm\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.645556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.645480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.745922 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.745876    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.745922 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.745924    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.745946    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.745986    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746020    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746050    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746091    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746115    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746141    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746174    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746199    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746228    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746253    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746286    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746313    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746351    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746395    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746556 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746456    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcbm\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-kube-api-access-fbcbm\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.746972 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.746913    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.747341 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.747120    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749181    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749243    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749271    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749349    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749725 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749577    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.749725 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749666    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.750146 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.749938    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.750146 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.750077    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.750277 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.750146    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.750408 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.750383    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.750768 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.750743    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f839d69a-1168-4d62-936d-a8c6bed97d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.751356 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.751334    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.751619 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.751602    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.751698 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.751678    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f839d69a-1168-4d62-936d-a8c6bed97d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.751776 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.751763    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f839d69a-1168-4d62-936d-a8c6bed97d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.754788 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.754770    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcbm\" (UniqueName: \"kubernetes.io/projected/f839d69a-1168-4d62-936d-a8c6bed97d6e-kube-api-access-fbcbm\") pod \"prometheus-k8s-0\" (UID: \"f839d69a-1168-4d62-936d-a8c6bed97d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.855249 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.855213    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:29:29.985573 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:29.985542    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 11:29:29.988977 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:29:29.988943    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf839d69a_1168_4d62_936d_a8c6bed97d6e.slice/crio-43641b70fc64a6731c55f160cdee26bb26f0375c25d82c15065f6637ed208c22 WatchSource:0}: Error finding container 43641b70fc64a6731c55f160cdee26bb26f0375c25d82c15065f6637ed208c22: Status 404 returned error can't find the container with id 43641b70fc64a6731c55f160cdee26bb26f0375c25d82c15065f6637ed208c22
May 26 11:29:30.484052 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:30.484018    2562 generic.go:358] "Generic (PLEG): container finished" podID="f839d69a-1168-4d62-936d-a8c6bed97d6e" containerID="580309f54db2e37699dab4e67e11c5c7efce9bc65d245e93aa1a9f0df5ba4b66" exitCode=0
May 26 11:29:30.484234 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:30.484105    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerDied","Data":"580309f54db2e37699dab4e67e11c5c7efce9bc65d245e93aa1a9f0df5ba4b66"}
May 26 11:29:30.484234 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:30.484143    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"43641b70fc64a6731c55f160cdee26bb26f0375c25d82c15065f6637ed208c22"}
May 26 11:29:30.682753 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:30.682304    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83" path="/var/lib/kubelet/pods/b8da6e5e-1a4e-43fa-ae8a-39071eb4eb83/volumes"
May 26 11:29:31.490665 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490635    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"e7456a3dce769cd5954553bf784ed2534e9c28b0e02999dfae8320303fc96289"}
May 26 11:29:31.490665 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490668    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"c6b2af973a35ab4414293bdc398aecb2caa2dee3ef677fca3562280f4777a85b"}
May 26 11:29:31.491038 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490678    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"3972c1b783180808388a856a400685298cc20d7f7b701d0745fc3306c4ec04e9"}
May 26 11:29:31.491038 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490686    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"5ca5b85e09a0afcfaa05d0c1ca21837fa1440ac60ed4aec6a2f2edf1992b93c0"}
May 26 11:29:31.491038 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490694    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"a5eb20603a0728c1ba505cd0e7c09f8bfd2493db10ad701a91e901647f7c6e04"}
May 26 11:29:31.491038 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.490701    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f839d69a-1168-4d62-936d-a8c6bed97d6e","Type":"ContainerStarted","Data":"74a15b46bf26913f667b51e67530da4c32e13690dfd361c182aa94f2474a8f3c"}
May 26 11:29:31.520878 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:31.520834    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.52081856 podStartE2EDuration="2.52081856s" podCreationTimestamp="2026-05-26 11:29:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:29:31.520019034 +0000 UTC m=+267.480566519" watchObservedRunningTime="2026-05-26 11:29:31.52081856 +0000 UTC m=+267.481366042"
May 26 11:29:34.856383 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:29:34.856345    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:30:04.558647 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:04.558615    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:30:04.562936 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:04.562913    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:30:04.565753 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:04.565734    2562 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:30:29.856330 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:29.856295    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:30:29.871583 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:29.871558    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:30:30.670309 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:30:30.670277    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 11:33:29.613460 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.613410    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z5xsh"]
May 26 11:33:29.616803 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.616775    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.619555 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.619535    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:33:29.672474 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.672445    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z5xsh"]
May 26 11:33:29.684618 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.684592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/217e3e9a-400f-4c89-aed7-d005b5a51e76-original-pull-secret\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.684758 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.684673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-dbus\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.684758 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.684724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-kubelet-config\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.785446 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.785394    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/217e3e9a-400f-4c89-aed7-d005b5a51e76-original-pull-secret\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.785605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.785470    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-dbus\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.785605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.785502    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-kubelet-config\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.785605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.785568    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-kubelet-config\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.785711 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.785677    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/217e3e9a-400f-4c89-aed7-d005b5a51e76-dbus\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.787742 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.787714    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/217e3e9a-400f-4c89-aed7-d005b5a51e76-original-pull-secret\") pod \"global-pull-secret-syncer-z5xsh\" (UID: \"217e3e9a-400f-4c89-aed7-d005b5a51e76\") " pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:29.926200 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:29.926170    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5xsh"
May 26 11:33:30.059185 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:30.059149    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z5xsh"]
May 26 11:33:30.063213 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:33:30.063186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217e3e9a_400f_4c89_aed7_d005b5a51e76.slice/crio-b196dc623d93aee7c617541dabcf7078378f81a5b15306d0d3c429711d11f83e WatchSource:0}: Error finding container b196dc623d93aee7c617541dabcf7078378f81a5b15306d0d3c429711d11f83e: Status 404 returned error can't find the container with id b196dc623d93aee7c617541dabcf7078378f81a5b15306d0d3c429711d11f83e
May 26 11:33:30.064783 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:30.064767    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:33:30.135339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:30.135311    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z5xsh" event={"ID":"217e3e9a-400f-4c89-aed7-d005b5a51e76","Type":"ContainerStarted","Data":"b196dc623d93aee7c617541dabcf7078378f81a5b15306d0d3c429711d11f83e"}
May 26 11:33:34.149785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:34.149747    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z5xsh" event={"ID":"217e3e9a-400f-4c89-aed7-d005b5a51e76","Type":"ContainerStarted","Data":"e9e4809539ce264fdd6c17821197cd62b761a61619f3129f65c14a4fc7cc3d52"}
May 26 11:33:34.174289 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:33:34.174234    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z5xsh" podStartSLOduration=1.841205404 podStartE2EDuration="5.174220647s" podCreationTimestamp="2026-05-26 11:33:29 +0000 UTC" firstStartedPulling="2026-05-26 11:33:30.064943225 +0000 UTC m=+506.025490688" lastFinishedPulling="2026-05-26 11:33:33.397958456 +0000 UTC m=+509.358505931" observedRunningTime="2026-05-26 11:33:34.174064755 +0000 UTC m=+510.134612240" watchObservedRunningTime="2026-05-26 11:33:34.174220647 +0000 UTC m=+510.134768132"
May 26 11:35:04.581075 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:35:04.581044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:35:04.581816 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:35:04.581794    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:36:23.340877 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.340838    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"]
May 26 11:36:23.343955 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.343937    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.346224 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.346197    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 26 11:36:23.346357 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.346291    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xth68\""
May 26 11:36:23.347013 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.346998    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 26 11:36:23.357778 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.357754    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"]
May 26 11:36:23.362835 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.362807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxkx\" (UniqueName: \"kubernetes.io/projected/95654c47-2165-41e2-9a98-5944bd332d03-kube-api-access-2qxkx\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.362926 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.362856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95654c47-2165-41e2-9a98-5944bd332d03-tmp\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.463465 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.463404    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxkx\" (UniqueName: \"kubernetes.io/projected/95654c47-2165-41e2-9a98-5944bd332d03-kube-api-access-2qxkx\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.463633 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.463473    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95654c47-2165-41e2-9a98-5944bd332d03-tmp\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.463836 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.463820    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95654c47-2165-41e2-9a98-5944bd332d03-tmp\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.472427 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.472390    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxkx\" (UniqueName: \"kubernetes.io/projected/95654c47-2165-41e2-9a98-5944bd332d03-kube-api-access-2qxkx\") pod \"openshift-lws-operator-bfc7f696d-zcxr5\" (UID: \"95654c47-2165-41e2-9a98-5944bd332d03\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.653286 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.653257    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"
May 26 11:36:23.780090 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:23.780054    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5"]
May 26 11:36:23.783319 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:36:23.783293    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95654c47_2165_41e2_9a98_5944bd332d03.slice/crio-787ab9f9f995e7da7b98998ac566cee0782bf652350f9d91f78a8c1030219427 WatchSource:0}: Error finding container 787ab9f9f995e7da7b98998ac566cee0782bf652350f9d91f78a8c1030219427: Status 404 returned error can't find the container with id 787ab9f9f995e7da7b98998ac566cee0782bf652350f9d91f78a8c1030219427
May 26 11:36:24.620695 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:24.620655    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5" event={"ID":"95654c47-2165-41e2-9a98-5944bd332d03","Type":"ContainerStarted","Data":"787ab9f9f995e7da7b98998ac566cee0782bf652350f9d91f78a8c1030219427"}
May 26 11:36:28.633498 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:28.633457    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5" event={"ID":"95654c47-2165-41e2-9a98-5944bd332d03","Type":"ContainerStarted","Data":"a1028ce44516c718f396be538b38cff8a7a357158a93d94365b26c4323fd1d60"}
May 26 11:36:28.651809 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:28.651756    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-zcxr5" podStartSLOduration=1.600262786 podStartE2EDuration="5.651741844s" podCreationTimestamp="2026-05-26 11:36:23 +0000 UTC" firstStartedPulling="2026-05-26 11:36:23.785075615 +0000 UTC m=+679.745623079" lastFinishedPulling="2026-05-26 11:36:27.836554675 +0000 UTC m=+683.797102137" observedRunningTime="2026-05-26 11:36:28.650626482 +0000 UTC m=+684.611173969" watchObservedRunningTime="2026-05-26 11:36:28.651741844 +0000 UTC m=+684.612289329"
May 26 11:36:42.004872 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.004831    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"]
May 26 11:36:42.008025 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.008010    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.013168 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.013144    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 26 11:36:42.013834 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.013781    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 26 11:36:42.013834 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.013817    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 26 11:36:42.014000 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.013838    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9rt5d\""
May 26 11:36:42.044969 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.044943    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"]
May 26 11:36:42.116571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.116528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-metrics-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.116571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.116572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26678c67-074c-4929-80ed-2f6b04c5575a-manager-config\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.116788 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.116596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.116788 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.116653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc72f\" (UniqueName: \"kubernetes.io/projected/26678c67-074c-4929-80ed-2f6b04c5575a-kube-api-access-qc72f\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.217959 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.217921    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-metrics-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.217959 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.217961    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26678c67-074c-4929-80ed-2f6b04c5575a-manager-config\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.218150 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.218090    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.218150 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.218130    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qc72f\" (UniqueName: \"kubernetes.io/projected/26678c67-074c-4929-80ed-2f6b04c5575a-kube-api-access-qc72f\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.218670 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.218650    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26678c67-074c-4929-80ed-2f6b04c5575a-manager-config\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.220380 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.220358    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-metrics-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.220493 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.220437    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26678c67-074c-4929-80ed-2f6b04c5575a-cert\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.227761 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.227740    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc72f\" (UniqueName: \"kubernetes.io/projected/26678c67-074c-4929-80ed-2f6b04c5575a-kube-api-access-qc72f\") pod \"lws-controller-manager-5b8db67594-pmh7r\" (UID: \"26678c67-074c-4929-80ed-2f6b04c5575a\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.316871 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.316776    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:42.442780 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.442739    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"]
May 26 11:36:42.447709 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:36:42.447682    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26678c67_074c_4929_80ed_2f6b04c5575a.slice/crio-f55d19edeb4fb4fd829c851ba1342ab7ed103022bf2fde0ca3413ddc7ac42818 WatchSource:0}: Error finding container f55d19edeb4fb4fd829c851ba1342ab7ed103022bf2fde0ca3413ddc7ac42818: Status 404 returned error can't find the container with id f55d19edeb4fb4fd829c851ba1342ab7ed103022bf2fde0ca3413ddc7ac42818
May 26 11:36:42.670578 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:42.670546    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r" event={"ID":"26678c67-074c-4929-80ed-2f6b04c5575a","Type":"ContainerStarted","Data":"f55d19edeb4fb4fd829c851ba1342ab7ed103022bf2fde0ca3413ddc7ac42818"}
May 26 11:36:44.678165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:44.678136    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r" event={"ID":"26678c67-074c-4929-80ed-2f6b04c5575a","Type":"ContainerStarted","Data":"8a10a5a99a75c0efe2cca0fc4e13574dbbd94e043f94c3f74593240cf93ea2ca"}
May 26 11:36:44.678165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:44.678169    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:36:44.711646 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:44.711587    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r" podStartSLOduration=2.01201079 podStartE2EDuration="3.711571743s" podCreationTimestamp="2026-05-26 11:36:41 +0000 UTC" firstStartedPulling="2026-05-26 11:36:42.449493069 +0000 UTC m=+698.410040532" lastFinishedPulling="2026-05-26 11:36:44.149054008 +0000 UTC m=+700.109601485" observedRunningTime="2026-05-26 11:36:44.710203101 +0000 UTC m=+700.670750587" watchObservedRunningTime="2026-05-26 11:36:44.711571743 +0000 UTC m=+700.672119227"
May 26 11:36:55.682839 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:36:55.682761    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-pmh7r"
May 26 11:37:28.352139 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.352096    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"]
May 26 11:37:28.355242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.355225    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:28.358192 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.358170    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 26 11:37:28.358305 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.358208    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 26 11:37:28.358867 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.358851    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
May 26 11:37:28.358959 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.358870    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-z2vnh\""
May 26 11:37:28.366880 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.366857    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"]
May 26 11:37:28.382321 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.382300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l54n\" (UniqueName: \"kubernetes.io/projected/23ec7f8a-039e-49d5-b67a-5156bd10c9b2-kube-api-access-8l54n\") pod \"dns-operator-controller-manager-844548ff4c-djxwj\" (UID: \"23ec7f8a-039e-49d5-b67a-5156bd10c9b2\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:28.483586 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.483556    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8l54n\" (UniqueName: \"kubernetes.io/projected/23ec7f8a-039e-49d5-b67a-5156bd10c9b2-kube-api-access-8l54n\") pod \"dns-operator-controller-manager-844548ff4c-djxwj\" (UID: \"23ec7f8a-039e-49d5-b67a-5156bd10c9b2\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:28.499347 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.499314    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l54n\" (UniqueName: \"kubernetes.io/projected/23ec7f8a-039e-49d5-b67a-5156bd10c9b2-kube-api-access-8l54n\") pod \"dns-operator-controller-manager-844548ff4c-djxwj\" (UID: \"23ec7f8a-039e-49d5-b67a-5156bd10c9b2\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:28.665297 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.665258    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:28.809427 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:28.809381    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"]
May 26 11:37:28.814234 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:37:28.814206    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ec7f8a_039e_49d5_b67a_5156bd10c9b2.slice/crio-0cee925087231e064fb60fef9a2ff59d461bec52fd4c5165b21ffd107ade4660 WatchSource:0}: Error finding container 0cee925087231e064fb60fef9a2ff59d461bec52fd4c5165b21ffd107ade4660: Status 404 returned error can't find the container with id 0cee925087231e064fb60fef9a2ff59d461bec52fd4c5165b21ffd107ade4660
May 26 11:37:29.800962 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:29.800906    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj" event={"ID":"23ec7f8a-039e-49d5-b67a-5156bd10c9b2","Type":"ContainerStarted","Data":"0cee925087231e064fb60fef9a2ff59d461bec52fd4c5165b21ffd107ade4660"}
May 26 11:37:30.580752 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.580716    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"]
May 26 11:37:30.589646 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.589616    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:30.592391 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.592370    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-q628l\""
May 26 11:37:30.600022 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.599992    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"]
May 26 11:37:30.702664 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.702625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfzd\" (UniqueName: \"kubernetes.io/projected/9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a-kube-api-access-wjfzd\") pod \"limitador-operator-controller-manager-c7fb4c8d5-892hk\" (UID: \"9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:30.803899 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.803857    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfzd\" (UniqueName: \"kubernetes.io/projected/9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a-kube-api-access-wjfzd\") pod \"limitador-operator-controller-manager-c7fb4c8d5-892hk\" (UID: \"9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:30.821013 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.820983    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfzd\" (UniqueName: \"kubernetes.io/projected/9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a-kube-api-access-wjfzd\") pod \"limitador-operator-controller-manager-c7fb4c8d5-892hk\" (UID: \"9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:30.902312 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:30.902281    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:31.494888 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:31.494860    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"]
May 26 11:37:31.497046 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:37:31.497016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4e27d7_09ed_49d3_8d4e_f29d09b3c48a.slice/crio-61728f7fe385c78f2e1c5212d9ee597cc2a09906ea24683266f9981c28077aa9 WatchSource:0}: Error finding container 61728f7fe385c78f2e1c5212d9ee597cc2a09906ea24683266f9981c28077aa9: Status 404 returned error can't find the container with id 61728f7fe385c78f2e1c5212d9ee597cc2a09906ea24683266f9981c28077aa9
May 26 11:37:31.807852 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:31.807814    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk" event={"ID":"9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a","Type":"ContainerStarted","Data":"61728f7fe385c78f2e1c5212d9ee597cc2a09906ea24683266f9981c28077aa9"}
May 26 11:37:31.809268 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:31.809242    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj" event={"ID":"23ec7f8a-039e-49d5-b67a-5156bd10c9b2","Type":"ContainerStarted","Data":"28a39daa661318055048a2b2d4af4db35f18a377816c37e0dac8c9ab1b926d15"}
May 26 11:37:31.809382 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:31.809360    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:31.832250 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:31.832200    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj" podStartSLOduration=1.225093615 podStartE2EDuration="3.832185211s" podCreationTimestamp="2026-05-26 11:37:28 +0000 UTC" firstStartedPulling="2026-05-26 11:37:28.816171863 +0000 UTC m=+744.776719326" lastFinishedPulling="2026-05-26 11:37:31.423263455 +0000 UTC m=+747.383810922" observedRunningTime="2026-05-26 11:37:31.830614866 +0000 UTC m=+747.791162351" watchObservedRunningTime="2026-05-26 11:37:31.832185211 +0000 UTC m=+747.792732697"
May 26 11:37:33.815941 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:33.815907    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk" event={"ID":"9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a","Type":"ContainerStarted","Data":"e76169e0ac01dc13fcc4ed321c08bd8a17cc68ce8792ac879b3303d91a88830b"}
May 26 11:37:33.816366 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:33.816020    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:37:33.834108 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:33.834060    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk" podStartSLOduration=2.263026238 podStartE2EDuration="3.834044819s" podCreationTimestamp="2026-05-26 11:37:30 +0000 UTC" firstStartedPulling="2026-05-26 11:37:31.499098957 +0000 UTC m=+747.459646421" lastFinishedPulling="2026-05-26 11:37:33.070117534 +0000 UTC m=+749.030665002" observedRunningTime="2026-05-26 11:37:33.832231052 +0000 UTC m=+749.792778561" watchObservedRunningTime="2026-05-26 11:37:33.834044819 +0000 UTC m=+749.794592305"
May 26 11:37:42.814398 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:42.814360    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-djxwj"
May 26 11:37:44.820883 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:37:44.820856    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-892hk"
May 26 11:38:18.218063 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.217981    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:18.224222 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.224199    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:18.228348 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.228327    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bb9gn\""
May 26 11:38:18.243319 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.243295    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:18.298536 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.298511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktr7\" (UniqueName: \"kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7\") pod \"authorino-674b59b84c-4ks86\" (UID: \"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1\") " pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:18.399863 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.399822    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nktr7\" (UniqueName: \"kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7\") pod \"authorino-674b59b84c-4ks86\" (UID: \"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1\") " pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:18.412097 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.412068    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktr7\" (UniqueName: \"kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7\") pod \"authorino-674b59b84c-4ks86\" (UID: \"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1\") " pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:18.533554 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.533464    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:18.655081 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.655037    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:18.657969 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:38:18.657940    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e4e615f_c7ad_49fe_96ed_fa593ac7fab1.slice/crio-a0b1b96d78a7798b73d03f1732fa56197b10ab9ca32b2df713aa1edc93849005 WatchSource:0}: Error finding container a0b1b96d78a7798b73d03f1732fa56197b10ab9ca32b2df713aa1edc93849005: Status 404 returned error can't find the container with id a0b1b96d78a7798b73d03f1732fa56197b10ab9ca32b2df713aa1edc93849005
May 26 11:38:18.951085 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:18.951048    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-4ks86" event={"ID":"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1","Type":"ContainerStarted","Data":"a0b1b96d78a7798b73d03f1732fa56197b10ab9ca32b2df713aa1edc93849005"}
May 26 11:38:21.962165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:21.962124    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-4ks86" event={"ID":"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1","Type":"ContainerStarted","Data":"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"}
May 26 11:38:21.977562 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:21.977515    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-4ks86" podStartSLOduration=1.344344423 podStartE2EDuration="3.977500767s" podCreationTimestamp="2026-05-26 11:38:18 +0000 UTC" firstStartedPulling="2026-05-26 11:38:18.659615994 +0000 UTC m=+794.620163457" lastFinishedPulling="2026-05-26 11:38:21.292772338 +0000 UTC m=+797.253319801" observedRunningTime="2026-05-26 11:38:21.975657583 +0000 UTC m=+797.936205079" watchObservedRunningTime="2026-05-26 11:38:21.977500767 +0000 UTC m=+797.938048251"
May 26 11:38:22.421073 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:22.421035    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:23.967296 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:23.967252    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-4ks86" podUID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" containerName="authorino" containerID="cri-o://30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd" gracePeriod=30
May 26 11:38:24.210407 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.210384    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:24.252524 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.252445    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nktr7\" (UniqueName: \"kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7\") pod \"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1\" (UID: \"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1\") "
May 26 11:38:24.254596 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.254567    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7" (OuterVolumeSpecName: "kube-api-access-nktr7") pod "0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" (UID: "0e4e615f-c7ad-49fe-96ed-fa593ac7fab1"). InnerVolumeSpecName "kube-api-access-nktr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:38:24.353021 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.352983    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktr7\" (UniqueName: \"kubernetes.io/projected/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1-kube-api-access-nktr7\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:38:24.970630 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.970542    2562 generic.go:358] "Generic (PLEG): container finished" podID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" containerID="30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd" exitCode=0
May 26 11:38:24.970630 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.970589    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-4ks86"
May 26 11:38:24.970630 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.970619    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-4ks86" event={"ID":"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1","Type":"ContainerDied","Data":"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"}
May 26 11:38:24.971063 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.970654    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-4ks86" event={"ID":"0e4e615f-c7ad-49fe-96ed-fa593ac7fab1","Type":"ContainerDied","Data":"a0b1b96d78a7798b73d03f1732fa56197b10ab9ca32b2df713aa1edc93849005"}
May 26 11:38:24.971063 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.970669    2562 scope.go:117] "RemoveContainer" containerID="30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"
May 26 11:38:24.978206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.978186    2562 scope.go:117] "RemoveContainer" containerID="30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"
May 26 11:38:24.978515 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:38:24.978489    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd\": container with ID starting with 30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd not found: ID does not exist" containerID="30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"
May 26 11:38:24.978594 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.978523    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd"} err="failed to get container status \"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd\": rpc error: code = NotFound desc = could not find container \"30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd\": container with ID starting with 30f7021abba6ad97587b459b4fb4377fb0520d03e75233b2ad9d8ef6f2c9becd not found: ID does not exist"
May 26 11:38:24.987297 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.987272    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:24.991014 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:24.990995    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-4ks86"]
May 26 11:38:26.678554 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:26.678520    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" path="/var/lib/kubelet/pods/0e4e615f-c7ad-49fe-96ed-fa593ac7fab1/volumes"
May 26 11:38:40.605041 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.605004    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-4bgz6"]
May 26 11:38:40.605561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.605320    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" containerName="authorino"
May 26 11:38:40.605561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.605331    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" containerName="authorino"
May 26 11:38:40.605561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.605390    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e4e615f-c7ad-49fe-96ed-fa593ac7fab1" containerName="authorino"
May 26 11:38:40.609706 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.609690    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.613205 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.613183    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bb9gn\""
May 26 11:38:40.613205 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.613200    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 26 11:38:40.617959 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.617936    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-4bgz6"]
May 26 11:38:40.688055 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.688021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrd6\" (UniqueName: \"kubernetes.io/projected/a9cf80bb-d98c-494b-9445-ce491eaa5e32-kube-api-access-vqrd6\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.688203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.688067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9cf80bb-d98c-494b-9445-ce491eaa5e32-tls-cert\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.789152 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.789110    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrd6\" (UniqueName: \"kubernetes.io/projected/a9cf80bb-d98c-494b-9445-ce491eaa5e32-kube-api-access-vqrd6\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.789334 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.789270    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9cf80bb-d98c-494b-9445-ce491eaa5e32-tls-cert\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.791683 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.791665    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9cf80bb-d98c-494b-9445-ce491eaa5e32-tls-cert\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.797282 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.797252    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrd6\" (UniqueName: \"kubernetes.io/projected/a9cf80bb-d98c-494b-9445-ce491eaa5e32-kube-api-access-vqrd6\") pod \"authorino-68bd676465-4bgz6\" (UID: \"a9cf80bb-d98c-494b-9445-ce491eaa5e32\") " pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:40.919354 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:40.919310    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-4bgz6"
May 26 11:38:41.045434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:41.045255    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-4bgz6"]
May 26 11:38:41.050667 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:38:41.050638    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9cf80bb_d98c_494b_9445_ce491eaa5e32.slice/crio-2f2de3e59414dcb486b95805971a449553fa7abad62f64a6ff69f1fb479f48c9 WatchSource:0}: Error finding container 2f2de3e59414dcb486b95805971a449553fa7abad62f64a6ff69f1fb479f48c9: Status 404 returned error can't find the container with id 2f2de3e59414dcb486b95805971a449553fa7abad62f64a6ff69f1fb479f48c9
May 26 11:38:41.051565 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:41.051545    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:38:42.023079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:42.023046    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-4bgz6" event={"ID":"a9cf80bb-d98c-494b-9445-ce491eaa5e32","Type":"ContainerStarted","Data":"bbfcb06a279cae561d843b78cc587ade555bbd0fea0fff3fb55c9a41e21b0a36"}
May 26 11:38:42.023480 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:42.023085    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-4bgz6" event={"ID":"a9cf80bb-d98c-494b-9445-ce491eaa5e32","Type":"ContainerStarted","Data":"2f2de3e59414dcb486b95805971a449553fa7abad62f64a6ff69f1fb479f48c9"}
May 26 11:38:42.040499 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:38:42.040451    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-4bgz6" podStartSLOduration=1.572582445 podStartE2EDuration="2.04043621s" podCreationTimestamp="2026-05-26 11:38:40 +0000 UTC" firstStartedPulling="2026-05-26 11:38:41.05167813 +0000 UTC m=+817.012225594" lastFinishedPulling="2026-05-26 11:38:41.519531897 +0000 UTC m=+817.480079359" observedRunningTime="2026-05-26 11:38:42.038797728 +0000 UTC m=+817.999345215" watchObservedRunningTime="2026-05-26 11:38:42.04043621 +0000 UTC m=+818.000983696"
May 26 11:39:01.228097 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.228059    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-pmqpb"]
May 26 11:39:01.234684 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.234662    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.238179 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.238159    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:39:01.238289 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.238231    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:39:01.238991 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.238972    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jbblx\""
May 26 11:39:01.239210 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.239084    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:39:01.242146 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.242102    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-pmqpb"]
May 26 11:39:01.357201 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.357161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/803ead6e-08fb-456a-9779-f36022f916a7-data\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.357389 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.357225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnff\" (UniqueName: \"kubernetes.io/projected/803ead6e-08fb-456a-9779-f36022f916a7-kube-api-access-xjnff\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.458218 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.458185    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnff\" (UniqueName: \"kubernetes.io/projected/803ead6e-08fb-456a-9779-f36022f916a7-kube-api-access-xjnff\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.458368 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.458264    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/803ead6e-08fb-456a-9779-f36022f916a7-data\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.458629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.458613    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/803ead6e-08fb-456a-9779-f36022f916a7-data\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.467220 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.467201    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnff\" (UniqueName: \"kubernetes.io/projected/803ead6e-08fb-456a-9779-f36022f916a7-kube-api-access-xjnff\") pod \"seaweedfs-79f457656b-pmqpb\" (UID: \"803ead6e-08fb-456a-9779-f36022f916a7\") " pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.545165 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.545075    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:01.669249 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:01.669216    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-pmqpb"]
May 26 11:39:01.672037 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:39:01.672009    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803ead6e_08fb_456a_9779_f36022f916a7.slice/crio-fb0dbf16e34c5fefed6e7b9fb76b91d88db74a49eb074540144a703e9c8f52dc WatchSource:0}: Error finding container fb0dbf16e34c5fefed6e7b9fb76b91d88db74a49eb074540144a703e9c8f52dc: Status 404 returned error can't find the container with id fb0dbf16e34c5fefed6e7b9fb76b91d88db74a49eb074540144a703e9c8f52dc
May 26 11:39:02.085750 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:02.085707    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-pmqpb" event={"ID":"803ead6e-08fb-456a-9779-f36022f916a7","Type":"ContainerStarted","Data":"fb0dbf16e34c5fefed6e7b9fb76b91d88db74a49eb074540144a703e9c8f52dc"}
May 26 11:39:05.529013 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:05.528991    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:39:06.101253 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:06.101214    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-pmqpb" event={"ID":"803ead6e-08fb-456a-9779-f36022f916a7","Type":"ContainerStarted","Data":"f6c4267a2d260e8f2895f53050cb2d221c86b705f218de34c0399e1e6e6c8c91"}
May 26 11:39:06.101466 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:06.101281    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:39:06.118194 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:06.118143    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-pmqpb" podStartSLOduration=1.2650245039999999 podStartE2EDuration="5.118128065s" podCreationTimestamp="2026-05-26 11:39:01 +0000 UTC" firstStartedPulling="2026-05-26 11:39:01.673590861 +0000 UTC m=+837.634138359" lastFinishedPulling="2026-05-26 11:39:05.526694453 +0000 UTC m=+841.487241920" observedRunningTime="2026-05-26 11:39:06.116783901 +0000 UTC m=+842.077331399" watchObservedRunningTime="2026-05-26 11:39:06.118128065 +0000 UTC m=+842.078675552"
May 26 11:39:12.107281 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:39:12.107243    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-pmqpb"
May 26 11:40:04.605830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:04.605799    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:40:04.606684 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:04.606664    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:40:13.588044 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.588004    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-f6bj8"]
May 26 11:40:13.591277 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.591258    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.594317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.594296    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 11:40:13.594463 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.594447    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-64d6n\""
May 26 11:40:13.606710 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.606687    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-f6bj8"]
May 26 11:40:13.656058 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.656018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdkg\" (UniqueName: \"kubernetes.io/projected/45e57a4d-3e30-4914-b0be-337c1650bf2e-kube-api-access-jgdkg\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.656225 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.656142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45e57a4d-3e30-4914-b0be-337c1650bf2e-cert\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.757051 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.757021    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45e57a4d-3e30-4914-b0be-337c1650bf2e-cert\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.757235 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.757078    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdkg\" (UniqueName: \"kubernetes.io/projected/45e57a4d-3e30-4914-b0be-337c1650bf2e-kube-api-access-jgdkg\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.759535 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.759512    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45e57a4d-3e30-4914-b0be-337c1650bf2e-cert\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.764943 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.764917    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdkg\" (UniqueName: \"kubernetes.io/projected/45e57a4d-3e30-4914-b0be-337c1650bf2e-kube-api-access-jgdkg\") pod \"odh-model-controller-696fc77849-f6bj8\" (UID: \"45e57a4d-3e30-4914-b0be-337c1650bf2e\") " pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:13.901881 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:13.901852    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:14.024721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:14.024691    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-f6bj8"]
May 26 11:40:14.027714 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:40:14.027680    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e57a4d_3e30_4914_b0be_337c1650bf2e.slice/crio-c85cfc8a68a1034f0c4ed455d8e551420f6bc4ed82d38ebfd6e6151b2d0dacd1 WatchSource:0}: Error finding container c85cfc8a68a1034f0c4ed455d8e551420f6bc4ed82d38ebfd6e6151b2d0dacd1: Status 404 returned error can't find the container with id c85cfc8a68a1034f0c4ed455d8e551420f6bc4ed82d38ebfd6e6151b2d0dacd1
May 26 11:40:14.315897 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:14.315805    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-f6bj8" event={"ID":"45e57a4d-3e30-4914-b0be-337c1650bf2e","Type":"ContainerStarted","Data":"c85cfc8a68a1034f0c4ed455d8e551420f6bc4ed82d38ebfd6e6151b2d0dacd1"}
May 26 11:40:17.328734 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:17.328697    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-f6bj8" event={"ID":"45e57a4d-3e30-4914-b0be-337c1650bf2e","Type":"ContainerStarted","Data":"45c570a921a55926623823bb02e76e8ca77d4cf47436907d73632085954b9a0a"}
May 26 11:40:17.329089 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:17.328857    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:40:17.348269 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:17.348222    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-f6bj8" podStartSLOduration=1.869701172 podStartE2EDuration="4.348207922s" podCreationTimestamp="2026-05-26 11:40:13 +0000 UTC" firstStartedPulling="2026-05-26 11:40:14.028982976 +0000 UTC m=+909.989530440" lastFinishedPulling="2026-05-26 11:40:16.507489723 +0000 UTC m=+912.468037190" observedRunningTime="2026-05-26 11:40:17.346658071 +0000 UTC m=+913.307205556" watchObservedRunningTime="2026-05-26 11:40:17.348207922 +0000 UTC m=+913.308755407"
May 26 11:40:28.334919 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:40:28.334886    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-f6bj8"
May 26 11:41:04.926677 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.926641    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:04.930049 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.930032    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.934014 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.933981    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qjhlk\""
May 26 11:41:04.934014 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.934010    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 26 11:41:04.934187 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.934011    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:41:04.934187 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.934104    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:41:04.942464 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.942435    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:04.987232 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.987232 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.987232 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.998707 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkqv\" (UniqueName: \"kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.998707 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.998707 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:04.998707 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:04.987338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.088659 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088620    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.088833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088678    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.088833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088725    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.088833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088771    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkqv\" (UniqueName: \"kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.088833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088816    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089039 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088838    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089039 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.088872    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089141 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.089089    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089520 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.089223    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089520 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.089279    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.089629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.089526    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.090970 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.090937    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.091202 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.091186    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.117794 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.117766    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkqv\" (UniqueName: \"kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.241141 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.241059    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:05.380884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.380842    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:05.384242 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:41:05.384218    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052b2dd3_5d8a_4e2f_a451_df7fdd0d3116.slice/crio-8a655c15e89233f8dd963f5518018eb794d8771e089db15dfb16edabd592ca38 WatchSource:0}: Error finding container 8a655c15e89233f8dd963f5518018eb794d8771e089db15dfb16edabd592ca38: Status 404 returned error can't find the container with id 8a655c15e89233f8dd963f5518018eb794d8771e089db15dfb16edabd592ca38
May 26 11:41:05.488510 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:05.488472    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerStarted","Data":"8a655c15e89233f8dd963f5518018eb794d8771e089db15dfb16edabd592ca38"}
May 26 11:41:10.508945 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:10.508909    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerStarted","Data":"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"}
May 26 11:41:14.524388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:14.524354    2562 generic.go:358] "Generic (PLEG): container finished" podID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerID="be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82" exitCode=0
May 26 11:41:14.524773 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:14.524402    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerDied","Data":"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"}
May 26 11:41:16.532978 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:16.532940    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerStarted","Data":"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"}
May 26 11:41:16.553852 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:16.553799    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" podStartSLOduration=2.34600688 podStartE2EDuration="12.553784338s" podCreationTimestamp="2026-05-26 11:41:04 +0000 UTC" firstStartedPulling="2026-05-26 11:41:05.386402154 +0000 UTC m=+961.346949617" lastFinishedPulling="2026-05-26 11:41:15.594179609 +0000 UTC m=+971.554727075" observedRunningTime="2026-05-26 11:41:16.552379659 +0000 UTC m=+972.512927144" watchObservedRunningTime="2026-05-26 11:41:16.553784338 +0000 UTC m=+972.514331823"
May 26 11:41:21.781297 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.781265    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:41:21.799735 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.799704    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:41:21.799863 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.799827    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.802322 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.802301    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-8s7cx\""
May 26 11:41:21.802540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.802525    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 26 11:41:21.942613 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.942613 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8hz\" (UniqueName: \"kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.942818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.942818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.942818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:21.942818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:21.942816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043485 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043368    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043485 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043441    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043485 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043473    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043500    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043528    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043785 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043546    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8hz\" (UniqueName: \"kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043920 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043846    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043920 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043872    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.043987 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.043933    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.044028 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.044003    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.045910 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.045894    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.052359 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.052338    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8hz\" (UniqueName: \"kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.108755 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.108720    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:22.241794 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.241761    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:41:22.245123 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:41:22.245094    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2a705f_2c61_4bad_bf38_86ca875b5444.slice/crio-dfe657492860477d56d93aae79cb8ace44985c5f995bc39ee3d195a1bea73fdc WatchSource:0}: Error finding container dfe657492860477d56d93aae79cb8ace44985c5f995bc39ee3d195a1bea73fdc: Status 404 returned error can't find the container with id dfe657492860477d56d93aae79cb8ace44985c5f995bc39ee3d195a1bea73fdc
May 26 11:41:22.553516 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.553397    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerStarted","Data":"b905db1313ee5faed0fa8c22e7a5aa30afd5cb7ff55cdfb33531d0bd1ae48f24"}
May 26 11:41:22.553516 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:22.553466    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerStarted","Data":"dfe657492860477d56d93aae79cb8ace44985c5f995bc39ee3d195a1bea73fdc"}
May 26 11:41:23.014168 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.014131    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:23.014619 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.014501    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="main" containerID="cri-o://d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c" gracePeriod=30
May 26 11:41:23.282106 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.282082    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:23.356090 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356052    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356090 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356090    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356335 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356159    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356335 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356182    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356335 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356206    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzkqv\" (UniqueName: \"kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356354    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356372    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:41:23.356503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356399    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location\") pod \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\" (UID: \"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116\") "
May 26 11:41:23.356503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356472    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache" (OuterVolumeSpecName: "model-cache") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:41:23.356724 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356689    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home" (OuterVolumeSpecName: "home") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:41:23.356778 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356703    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.356778 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.356751    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.358600 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.358568    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv" (OuterVolumeSpecName: "kube-api-access-dzkqv") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "kube-api-access-dzkqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:41:23.358600 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.358570    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:41:23.358777 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.358642    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm" (OuterVolumeSpecName: "dshm") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:41:23.458054 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.458020    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.458054 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.458049    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.458244 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.458064    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkqv\" (UniqueName: \"kubernetes.io/projected/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kube-api-access-dzkqv\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.458244 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.458078    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.558915 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.558823    2562 generic.go:358] "Generic (PLEG): container finished" podID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerID="d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c" exitCode=0
May 26 11:41:23.558915 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.558897    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerDied","Data":"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"}
May 26 11:41:23.558915 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.558907    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"
May 26 11:41:23.559162 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.558933    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7" event={"ID":"052b2dd3-5d8a-4e2f-a451-df7fdd0d3116","Type":"ContainerDied","Data":"8a655c15e89233f8dd963f5518018eb794d8771e089db15dfb16edabd592ca38"}
May 26 11:41:23.559162 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.558955    2562 scope.go:117] "RemoveContainer" containerID="d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"
May 26 11:41:23.560268 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.560241    2562 generic.go:358] "Generic (PLEG): container finished" podID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerID="b905db1313ee5faed0fa8c22e7a5aa30afd5cb7ff55cdfb33531d0bd1ae48f24" exitCode=0
May 26 11:41:23.560389 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.560324    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerDied","Data":"b905db1313ee5faed0fa8c22e7a5aa30afd5cb7ff55cdfb33531d0bd1ae48f24"}
May 26 11:41:23.573867 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.573845    2562 scope.go:117] "RemoveContainer" containerID="be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"
May 26 11:41:23.638129 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.638105    2562 scope.go:117] "RemoveContainer" containerID="d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"
May 26 11:41:23.638494 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:41:23.638470    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c\": container with ID starting with d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c not found: ID does not exist" containerID="d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"
May 26 11:41:23.638558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.638503    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c"} err="failed to get container status \"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c\": rpc error: code = NotFound desc = could not find container \"d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c\": container with ID starting with d2a65d42af1bc1ee4b5721ab0173b95b0b1dc4cc06952f573d81a35ea40c531c not found: ID does not exist"
May 26 11:41:23.638558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.638524    2562 scope.go:117] "RemoveContainer" containerID="be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"
May 26 11:41:23.638813 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:41:23.638795    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82\": container with ID starting with be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82 not found: ID does not exist" containerID="be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"
May 26 11:41:23.638852 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.638823    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82"} err="failed to get container status \"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82\": rpc error: code = NotFound desc = could not find container \"be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82\": container with ID starting with be9f43ecf220a2271a46f95a28b7d1ae4312367b3ac2162d3d1639e286ab9a82 not found: ID does not exist"
May 26 11:41:23.750791 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.750742    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" (UID: "052b2dd3-5d8a-4e2f-a451-df7fdd0d3116"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:41:23.760761 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.760733    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:41:23.885188 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.885150    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:23.888933 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:23.888905    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-54d6b5444fnxlh7"]
May 26 11:41:24.678858 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:24.678830    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" path="/var/lib/kubelet/pods/052b2dd3-5d8a-4e2f-a451-df7fdd0d3116/volumes"
May 26 11:41:25.571496 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:25.571457    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerStarted","Data":"1c0cc11194205126f4e7b96550c33263bedfd656e031fe90ac012b642758169d"}
May 26 11:41:46.532770 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.532737    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:41:46.533317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.533202    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="main"
May 26 11:41:46.533317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.533220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="main"
May 26 11:41:46.533317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.533263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="storage-initializer"
May 26 11:41:46.533317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.533270    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="storage-initializer"
May 26 11:41:46.533564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.533363    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="052b2dd3-5d8a-4e2f-a451-df7fdd0d3116" containerName="main"
May 26 11:41:46.604484 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.604400    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:41:46.604645 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.604580    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.607158 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.607126    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
May 26 11:41:46.667522 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667694 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667694 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdsd\" (UniqueName: \"kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667694 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667694 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.667884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.667822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.768880 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.768841    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.768891    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.768919    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdsd\" (UniqueName: \"kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.768968    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769003    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769071 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769032    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769343 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769085    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769343 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769329    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769477 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769375    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769534 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769470    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.769632 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.769597    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.771856 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.771834    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.772243 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.772217    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.778287 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.778258    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdsd\" (UniqueName: \"kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:46.919478 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:46.919435    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:41:54.209747 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:54.209702    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:41:54.318111 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:41:54.318072    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027 WatchSource:0}: Error finding container 9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027: Status 404 returned error can't find the container with id 9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027
May 26 11:41:54.685194 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:54.685170    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m" event={"ID":"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4","Type":"ContainerStarted","Data":"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"}
May 26 11:41:54.685292 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:54.685200    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m" event={"ID":"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4","Type":"ContainerStarted","Data":"9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027"}
May 26 11:41:55.692289 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:55.692224    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerStarted","Data":"b80222378475627b69663a0399b8e26c698da55b137720e98e208e257b88b427"}
May 26 11:41:55.713058 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:55.712970    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" podStartSLOduration=3.616854825 podStartE2EDuration="34.712951152s" podCreationTimestamp="2026-05-26 11:41:21 +0000 UTC" firstStartedPulling="2026-05-26 11:41:23.561449042 +0000 UTC m=+979.521996504" lastFinishedPulling="2026-05-26 11:41:54.657545354 +0000 UTC m=+1010.618092831" observedRunningTime="2026-05-26 11:41:55.712644012 +0000 UTC m=+1011.673191502" watchObservedRunningTime="2026-05-26 11:41:55.712951152 +0000 UTC m=+1011.673498638"
May 26 11:41:56.695732 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:56.695699    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:56.698073 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:56.698049    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:41:57.817061 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:57.817024    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:41:57.817548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:41:57.817479    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m" podUID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" containerName="storage-initializer" containerID="cri-o://b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201" gracePeriod=30
May 26 11:42:02.109274 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:02.109237    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:42:02.109274 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:02.109282    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:42:02.109789 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:02.109541    2562 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.25:8082/healthz\": dial tcp 10.134.0.25:8082: connect: connection refused"
May 26 11:42:12.111331 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:12.111300    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:42:12.112464 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:12.112445    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:42:27.878794 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:42:27.878749    2562 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-conmon-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache]"
May 26 11:42:27.879123 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:42:27.878787    2562 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-conmon-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache]"
May 26 11:42:27.879123 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:42:27.878755    2562 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-conmon-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache]"
May 26 11:42:27.879123 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:42:27.878922    2562 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-conmon-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765e7d6_54f2_4d0b_a076_1ff7f63b63b4.slice/crio-b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201.scope\": RecentStats: unable to find data in memory cache]"
May 26 11:42:28.021666 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.021643    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m_7765e7d6-54f2-4d0b-a076-1ff7f63b63b4/storage-initializer/0.log"
May 26 11:42:28.021779 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.021705    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:42:28.044072 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044048    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044215 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044090    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044215 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044122    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044215 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044139    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044215 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044164    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044215 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044193    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzdsd\" (UniqueName: \"kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044496 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044237    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs\") pod \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\" (UID: \"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4\") "
May 26 11:42:28.044496 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044346    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:28.044496 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044359    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:28.044496 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044468    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home" (OuterVolumeSpecName: "home") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:28.044655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044540    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.044655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044561    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.044655 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.044580    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.046205 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.046179    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm" (OuterVolumeSpecName: "dshm") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:28.046304 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.046263    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd" (OuterVolumeSpecName: "kube-api-access-kzdsd") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "kube-api-access-kzdsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:42:28.046304 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.046290    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:42:28.070632 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.070606    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" (UID: "7765e7d6-54f2-4d0b-a076-1ff7f63b63b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:28.145066 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.145036    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.145066 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.145065    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.145272 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.145075    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdsd\" (UniqueName: \"kubernetes.io/projected/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-kube-api-access-kzdsd\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.145272 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.145084    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:42:28.801559 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801528    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m_7765e7d6-54f2-4d0b-a076-1ff7f63b63b4/storage-initializer/0.log"
May 26 11:42:28.801721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801569    2562 generic.go:358] "Generic (PLEG): container finished" podID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" containerID="b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201" exitCode=137
May 26 11:42:28.801721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801640    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"
May 26 11:42:28.801721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801654    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m" event={"ID":"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4","Type":"ContainerDied","Data":"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"}
May 26 11:42:28.801721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801690    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m" event={"ID":"7765e7d6-54f2-4d0b-a076-1ff7f63b63b4","Type":"ContainerDied","Data":"9c8f10112e6f881aca593e63ce562d04b09774be923bb031386f2db9d4c9c027"}
May 26 11:42:28.801721 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.801707    2562 scope.go:117] "RemoveContainer" containerID="b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"
May 26 11:42:28.833157 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.832726    2562 scope.go:117] "RemoveContainer" containerID="b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"
May 26 11:42:28.833321 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:42:28.833294    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201\": container with ID starting with b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201 not found: ID does not exist" containerID="b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"
May 26 11:42:28.833489 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.833331    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201"} err="failed to get container status \"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201\": rpc error: code = NotFound desc = could not find container \"b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201\": container with ID starting with b6db13da1d4c04689ae2965ab232dbf3d98344dfd5782b3e28a0653f35aa1201 not found: ID does not exist"
May 26 11:42:28.834657 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.834636    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:42:28.843610 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:28.843587    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5ff94db8f46nf6m"]
May 26 11:42:30.678553 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:30.678518    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" path="/var/lib/kubelet/pods/7765e7d6-54f2-4d0b-a076-1ff7f63b63b4/volumes"
May 26 11:42:42.586298 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.586259    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:42:42.586752 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.586622    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" containerName="storage-initializer"
May 26 11:42:42.586752 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.586636    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" containerName="storage-initializer"
May 26 11:42:42.586752 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.586694    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7765e7d6-54f2-4d0b-a076-1ff7f63b63b4" containerName="storage-initializer"
May 26 11:42:42.591699 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.591669    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.594103 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.594082    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-pdrc8\""
May 26 11:42:42.594629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.594611    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 26 11:42:42.601644 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.601622    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:42:42.675106 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.675281 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.675281 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58lm\" (UniqueName: \"kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.675281 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.675450 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.675450 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.675402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776606 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776571    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x58lm\" (UniqueName: \"kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776606 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776620    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776873 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776761    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776873 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776843    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776977 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776884    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.776977 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.776945    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.777173 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.777148    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.777252 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.777220    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.777310 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.777298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.777387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.777364    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.779088 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.779070    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.784588 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.784563    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58lm\" (UniqueName: \"kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:42.902200 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:42.902156    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:43.034634 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:43.034597    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:42:43.037716 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:42:43.037675    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca24a014_8fc3_49e8_bbd1_480466f10c83.slice/crio-506c42a73a9a9bc5115c0708cb96e8fa2a8d05ab6aa5cbdc1cfd4f81985fdf79 WatchSource:0}: Error finding container 506c42a73a9a9bc5115c0708cb96e8fa2a8d05ab6aa5cbdc1cfd4f81985fdf79: Status 404 returned error can't find the container with id 506c42a73a9a9bc5115c0708cb96e8fa2a8d05ab6aa5cbdc1cfd4f81985fdf79
May 26 11:42:43.853301 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:43.853262    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerID="eee32644c7bd7dd3d8ece116dc669090fb961c5aba3b711a0c75c126ad673dfe" exitCode=0
May 26 11:42:43.853656 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:43.853347    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerDied","Data":"eee32644c7bd7dd3d8ece116dc669090fb961c5aba3b711a0c75c126ad673dfe"}
May 26 11:42:43.853656 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:43.853383    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerStarted","Data":"506c42a73a9a9bc5115c0708cb96e8fa2a8d05ab6aa5cbdc1cfd4f81985fdf79"}
May 26 11:42:44.858346 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:44.858299    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerStarted","Data":"42055e5400d6e4b6cfdf51eae24344752806707ad711b226fc0d82446009214f"}
May 26 11:42:44.858346 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:44.858345    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerStarted","Data":"58628cdaf7ca857eaf56ac950687e595b000f0468f342d76e436daf51b906dfd"}
May 26 11:42:44.858888 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:44.858575    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:44.882247 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:44.882193    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" podStartSLOduration=2.882177166 podStartE2EDuration="2.882177166s" podCreationTimestamp="2026-05-26 11:42:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:42:44.880431981 +0000 UTC m=+1060.840979460" watchObservedRunningTime="2026-05-26 11:42:44.882177166 +0000 UTC m=+1060.842724651"
May 26 11:42:52.903348 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:52.903263    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:52.903348 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:52.903308    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:52.906001 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:52.905974    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:42:53.893101 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:42:53.893073    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:43:03.769389 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:03.769354    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:43:03.769930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:03.769676    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="main" containerID="cri-o://1c0cc11194205126f4e7b96550c33263bedfd656e031fe90ac012b642758169d" gracePeriod=30
May 26 11:43:03.769930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:03.769705    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="tokenizer" containerID="cri-o://b80222378475627b69663a0399b8e26c698da55b137720e98e208e257b88b427" gracePeriod=30
May 26 11:43:03.929629 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:03.929571    2562 generic.go:358] "Generic (PLEG): container finished" podID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerID="1c0cc11194205126f4e7b96550c33263bedfd656e031fe90ac012b642758169d" exitCode=0
May 26 11:43:03.929786 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:03.929647    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerDied","Data":"1c0cc11194205126f4e7b96550c33263bedfd656e031fe90ac012b642758169d"}
May 26 11:43:04.935209 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:04.935175    2562 generic.go:358] "Generic (PLEG): container finished" podID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerID="b80222378475627b69663a0399b8e26c698da55b137720e98e208e257b88b427" exitCode=0
May 26 11:43:04.935570 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:04.935233    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerDied","Data":"b80222378475627b69663a0399b8e26c698da55b137720e98e208e257b88b427"}
May 26 11:43:05.019676 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.019653    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:43:05.164529 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164493    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k8hz\" (UniqueName: \"kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164537    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164567    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164677    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164727 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164717    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164950 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164774    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs\") pod \"fb2a705f-2c61-4bad-bf38-86ca875b5444\" (UID: \"fb2a705f-2c61-4bad-bf38-86ca875b5444\") "
May 26 11:43:05.164950 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164843    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:05.164950 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.164915    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:05.165107 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.165086    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:05.165162 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.165115    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.165162 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.165136    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.165349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.165331    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:05.166826 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.166805    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:43:05.166908 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.166861    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz" (OuterVolumeSpecName: "kube-api-access-2k8hz") pod "fb2a705f-2c61-4bad-bf38-86ca875b5444" (UID: "fb2a705f-2c61-4bad-bf38-86ca875b5444"). InnerVolumeSpecName "kube-api-access-2k8hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:43:05.265715 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.265683    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.265715 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.265718    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2a705f-2c61-4bad-bf38-86ca875b5444-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.265901 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.265737    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8hz\" (UniqueName: \"kubernetes.io/projected/fb2a705f-2c61-4bad-bf38-86ca875b5444-kube-api-access-2k8hz\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.265901 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.265748    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2a705f-2c61-4bad-bf38-86ca875b5444-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:05.940537 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.940503    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl" event={"ID":"fb2a705f-2c61-4bad-bf38-86ca875b5444","Type":"ContainerDied","Data":"dfe657492860477d56d93aae79cb8ace44985c5f995bc39ee3d195a1bea73fdc"}
May 26 11:43:05.940936 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.940547    2562 scope.go:117] "RemoveContainer" containerID="b80222378475627b69663a0399b8e26c698da55b137720e98e208e257b88b427"
May 26 11:43:05.940936 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.940546    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"
May 26 11:43:05.949123 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.949104    2562 scope.go:117] "RemoveContainer" containerID="1c0cc11194205126f4e7b96550c33263bedfd656e031fe90ac012b642758169d"
May 26 11:43:05.956057 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.956040    2562 scope.go:117] "RemoveContainer" containerID="b905db1313ee5faed0fa8c22e7a5aa30afd5cb7ff55cdfb33531d0bd1ae48f24"
May 26 11:43:05.963687 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.963667    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:43:05.970026 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:05.970006    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-68966tn6wl"]
May 26 11:43:06.679503 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:06.679473    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" path="/var/lib/kubelet/pods/fb2a705f-2c61-4bad-bf38-86ca875b5444/volumes"
May 26 11:43:09.719776 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.719742    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720093    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="main"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720106    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="main"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="storage-initializer"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720128    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="storage-initializer"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720135    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="tokenizer"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="tokenizer"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720191    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="main"
May 26 11:43:09.720226 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.720201    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb2a705f-2c61-4bad-bf38-86ca875b5444" containerName="tokenizer"
May 26 11:43:09.723054 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.723036    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.725696 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.725672    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-cpdqt\""
May 26 11:43:09.725798 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.725719    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 26 11:43:09.737746 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.737727    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:09.907203 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.907374 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.907374 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.907374 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.907374 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgkz\" (UniqueName: \"kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:09.907569 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:09.907454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008183    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008231    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008266 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008252    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008277    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008297    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008324    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgkz\" (UniqueName: \"kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008731 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008663    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008731 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008701    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008764    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.008833 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.008772    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.010813 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.010781    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.016146 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.016123    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgkz\" (UniqueName: \"kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.032048 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.032023    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:10.170207 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.169923    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:10.173030 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:43:10.172989    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883f9044_a09b_458a_8c6d_03ab31138082.slice/crio-1bdf2ba58d22da0fa4e38845f16c8be950e9deacefcf98d4f519f1666c9ad5ab WatchSource:0}: Error finding container 1bdf2ba58d22da0fa4e38845f16c8be950e9deacefcf98d4f519f1666c9ad5ab: Status 404 returned error can't find the container with id 1bdf2ba58d22da0fa4e38845f16c8be950e9deacefcf98d4f519f1666c9ad5ab
May 26 11:43:10.959076 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.959040    2562 generic.go:358] "Generic (PLEG): container finished" podID="883f9044-a09b-458a-8c6d-03ab31138082" containerID="e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c" exitCode=0
May 26 11:43:10.959446 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.959080    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerDied","Data":"e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c"}
May 26 11:43:10.959446 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:10.959103    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerStarted","Data":"1bdf2ba58d22da0fa4e38845f16c8be950e9deacefcf98d4f519f1666c9ad5ab"}
May 26 11:43:11.964644 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:11.964612    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerStarted","Data":"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"}
May 26 11:43:11.964644 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:11.964645    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerStarted","Data":"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"}
May 26 11:43:11.965168 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:11.964731    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:11.991814 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:11.991763    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" podStartSLOduration=2.991746911 podStartE2EDuration="2.991746911s" podCreationTimestamp="2026-05-26 11:43:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:43:11.988606894 +0000 UTC m=+1087.949154380" watchObservedRunningTime="2026-05-26 11:43:11.991746911 +0000 UTC m=+1087.952294405"
May 26 11:43:14.897253 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:14.897225    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:43:20.032562 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:20.032529    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:20.032562 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:20.032568    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:20.035277 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:20.035252    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:20.996554 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:20.996529    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:42.000344 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:42.000313    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:43.431630 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:43.431593    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:43.432086 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:43.431909    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="main" containerID="cri-o://139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e" gracePeriod=30
May 26 11:43:43.432086 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:43.431968    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="tokenizer" containerID="cri-o://c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0" gracePeriod=30
May 26 11:43:44.076824 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.076792    2562 generic.go:358] "Generic (PLEG): container finished" podID="883f9044-a09b-458a-8c6d-03ab31138082" containerID="139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e" exitCode=0
May 26 11:43:44.076990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.076872    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerDied","Data":"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"}
May 26 11:43:44.677104 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.677079    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:44.813020 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.812924    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813020 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.812994    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813032    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813054    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813104    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pgkz\" (UniqueName: \"kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813153    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds\") pod \"883f9044-a09b-458a-8c6d-03ab31138082\" (UID: \"883f9044-a09b-458a-8c6d-03ab31138082\") "
May 26 11:43:44.813514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813275    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:44.813514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813500    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:44.813514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813498    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:44.813651 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813508    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:44.813811 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.813792    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:44.815228 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.815208    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz" (OuterVolumeSpecName: "kube-api-access-7pgkz") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "kube-api-access-7pgkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:43:44.815333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.815293    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "883f9044-a09b-458a-8c6d-03ab31138082" (UID: "883f9044-a09b-458a-8c6d-03ab31138082"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:43:44.913930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.913898    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:44.913930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.913926    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883f9044-a09b-458a-8c6d-03ab31138082-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:44.913930 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.913937    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:44.914151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.913946    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgkz\" (UniqueName: \"kubernetes.io/projected/883f9044-a09b-458a-8c6d-03ab31138082-kube-api-access-7pgkz\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:44.914151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:44.913955    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/883f9044-a09b-458a-8c6d-03ab31138082-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:43:45.081754 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.081672    2562 generic.go:358] "Generic (PLEG): container finished" podID="883f9044-a09b-458a-8c6d-03ab31138082" containerID="c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0" exitCode=0
May 26 11:43:45.081754 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.081733    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerDied","Data":"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"}
May 26 11:43:45.081754 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.081738    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"
May 26 11:43:45.081754 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.081757    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55" event={"ID":"883f9044-a09b-458a-8c6d-03ab31138082","Type":"ContainerDied","Data":"1bdf2ba58d22da0fa4e38845f16c8be950e9deacefcf98d4f519f1666c9ad5ab"}
May 26 11:43:45.081992 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.081772    2562 scope.go:117] "RemoveContainer" containerID="c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"
May 26 11:43:45.093341 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.093313    2562 scope.go:117] "RemoveContainer" containerID="139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"
May 26 11:43:45.100605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.100584    2562 scope.go:117] "RemoveContainer" containerID="e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c"
May 26 11:43:45.106640 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.106613    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:45.108036 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108021    2562 scope.go:117] "RemoveContainer" containerID="c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"
May 26 11:43:45.108293 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:43:45.108270    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0\": container with ID starting with c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0 not found: ID does not exist" containerID="c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"
May 26 11:43:45.108346 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108301    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0"} err="failed to get container status \"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0\": rpc error: code = NotFound desc = could not find container \"c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0\": container with ID starting with c133465415ac27585d588ea176069033dcd134cb54733bed25428b37c22376c0 not found: ID does not exist"
May 26 11:43:45.108346 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108319    2562 scope.go:117] "RemoveContainer" containerID="139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"
May 26 11:43:45.108604 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:43:45.108582    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e\": container with ID starting with 139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e not found: ID does not exist" containerID="139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"
May 26 11:43:45.108672 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108614    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e"} err="failed to get container status \"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e\": rpc error: code = NotFound desc = could not find container \"139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e\": container with ID starting with 139e7dc1ed5ca0acdea803e942d55ce82d6ffe50cfd93d2982277500794da90e not found: ID does not exist"
May 26 11:43:45.108672 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108635    2562 scope.go:117] "RemoveContainer" containerID="e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c"
May 26 11:43:45.108877 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:43:45.108850    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c\": container with ID starting with e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c not found: ID does not exist" containerID="e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c"
May 26 11:43:45.108965 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.108883    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c"} err="failed to get container status \"e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c\": rpc error: code = NotFound desc = could not find container \"e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c\": container with ID starting with e6cbaf522ee58b4d513631cad5fb5c7399402b983e2b6d4a756faebf50ecbc4c not found: ID does not exist"
May 26 11:43:45.111251 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:45.111223    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7db8dtqr55"]
May 26 11:43:46.678678 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:46.678648    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883f9044-a09b-458a-8c6d-03ab31138082" path="/var/lib/kubelet/pods/883f9044-a09b-458a-8c6d-03ab31138082/volumes"
May 26 11:43:54.724050 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724010    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724405    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="tokenizer"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724432    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="tokenizer"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724457    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="storage-initializer"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724465    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="storage-initializer"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724471    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="main"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724477    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="main"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724529    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="tokenizer"
May 26 11:43:54.724540 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.724541    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="883f9044-a09b-458a-8c6d-03ab31138082" containerName="main"
May 26 11:43:54.727512 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.727495    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.729967 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.729945    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-qh6t9\""
May 26 11:43:54.730078 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.730011    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 26 11:43:54.738845 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.738819    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:43:54.901805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.901777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.901805 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.901809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.902010 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.901828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg4d\" (UniqueName: \"kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.902010 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.901951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.902010 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.901977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:54.902162 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:54.902104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.002906 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002813    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.002906 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002854    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.002906 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002902    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003171 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002927    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003171 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002952    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003171 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.002969    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg4d\" (UniqueName: \"kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003378 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.003355    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003481 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.003399    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003481 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.003466    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.003559 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.003494    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.005288 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.005271    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.012633 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.012610    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg4d\" (UniqueName: \"kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.039248 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.039224    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:55.178106 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.178076    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:43:55.181600 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:43:55.181573    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb156354d_cbd4_4366_9e6d_5828959a86a6.slice/crio-0a3fa5923bf05d0f02faf2c3785661f9e63eaa0c8d4a1f30aeeb9dac34088e5c WatchSource:0}: Error finding container 0a3fa5923bf05d0f02faf2c3785661f9e63eaa0c8d4a1f30aeeb9dac34088e5c: Status 404 returned error can't find the container with id 0a3fa5923bf05d0f02faf2c3785661f9e63eaa0c8d4a1f30aeeb9dac34088e5c
May 26 11:43:55.183400 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:55.183386    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:43:56.119314 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:56.119278    2562 generic.go:358] "Generic (PLEG): container finished" podID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerID="9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f" exitCode=0
May 26 11:43:56.119695 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:56.119346    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerDied","Data":"9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f"}
May 26 11:43:56.119695 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:56.119369    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerStarted","Data":"0a3fa5923bf05d0f02faf2c3785661f9e63eaa0c8d4a1f30aeeb9dac34088e5c"}
May 26 11:43:57.125230 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:57.125191    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerStarted","Data":"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"}
May 26 11:43:57.125230 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:57.125229    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerStarted","Data":"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"}
May 26 11:43:57.125817 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:57.125367    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:43:57.148103 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:43:57.148055    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" podStartSLOduration=3.148036615 podStartE2EDuration="3.148036615s" podCreationTimestamp="2026-05-26 11:43:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:43:57.145555073 +0000 UTC m=+1133.106102558" watchObservedRunningTime="2026-05-26 11:43:57.148036615 +0000 UTC m=+1133.108584099"
May 26 11:44:05.040353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:44:05.040319    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:44:05.040353 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:44:05.040357    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:44:05.043050 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:44:05.043024    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:44:05.154517 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:44:05.154489    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:44:26.160804 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:44:26.160723    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:45:04.637595 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:04.637564    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:45:04.638659 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:04.638632    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:45:26.448388 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:26.448350    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:45:26.448889 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:26.448730    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="main" containerID="cri-o://cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e" gracePeriod=30
May 26 11:45:26.448889 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:26.448770    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="tokenizer" containerID="cri-o://d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785" gracePeriod=30
May 26 11:45:27.432764 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.432728    2562 generic.go:358] "Generic (PLEG): container finished" podID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerID="cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e" exitCode=0
May 26 11:45:27.432960 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.432802    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerDied","Data":"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"}
May 26 11:45:27.698057 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.698033    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:45:27.795493 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795463    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jg4d\" (UniqueName: \"kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795633 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795506    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795633 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795558    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795633 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795622    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795810 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795672    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795810 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795700    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location\") pod \"b156354d-cbd4-4366-9e6d-5828959a86a6\" (UID: \"b156354d-cbd4-4366-9e6d-5828959a86a6\") "
May 26 11:45:27.795908 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795811    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.795955 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795940    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.796007 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795974    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.796007 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.795986    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:27.796434 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.796400    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.797704 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.797677    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:27.797798 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.797765    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d" (OuterVolumeSpecName: "kube-api-access-2jg4d") pod "b156354d-cbd4-4366-9e6d-5828959a86a6" (UID: "b156354d-cbd4-4366-9e6d-5828959a86a6"). InnerVolumeSpecName "kube-api-access-2jg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:27.897405 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.897374    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b156354d-cbd4-4366-9e6d-5828959a86a6-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:27.897405 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.897399    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:27.897605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.897410    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:27.897605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.897443    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jg4d\" (UniqueName: \"kubernetes.io/projected/b156354d-cbd4-4366-9e6d-5828959a86a6-kube-api-access-2jg4d\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:27.897605 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:27.897452    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b156354d-cbd4-4366-9e6d-5828959a86a6-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:45:28.437762 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.437731    2562 generic.go:358] "Generic (PLEG): container finished" podID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerID="d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785" exitCode=0
May 26 11:45:28.437953 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.437810    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"
May 26 11:45:28.437953 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.437807    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerDied","Data":"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"}
May 26 11:45:28.437953 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.437926    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td" event={"ID":"b156354d-cbd4-4366-9e6d-5828959a86a6","Type":"ContainerDied","Data":"0a3fa5923bf05d0f02faf2c3785661f9e63eaa0c8d4a1f30aeeb9dac34088e5c"}
May 26 11:45:28.437953 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.437944    2562 scope.go:117] "RemoveContainer" containerID="d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"
May 26 11:45:28.446164 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.446146    2562 scope.go:117] "RemoveContainer" containerID="cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"
May 26 11:45:28.453106 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.453086    2562 scope.go:117] "RemoveContainer" containerID="9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f"
May 26 11:45:28.459927 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.459908    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:45:28.461117 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.461103    2562 scope.go:117] "RemoveContainer" containerID="d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"
May 26 11:45:28.461350 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:45:28.461335    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785\": container with ID starting with d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785 not found: ID does not exist" containerID="d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"
May 26 11:45:28.461397 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.461356    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785"} err="failed to get container status \"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785\": rpc error: code = NotFound desc = could not find container \"d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785\": container with ID starting with d3c81a642cd539d78a46fdf08042010819187c60073df7de4897c71727249785 not found: ID does not exist"
May 26 11:45:28.461397 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.461373    2562 scope.go:117] "RemoveContainer" containerID="cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"
May 26 11:45:28.461723 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:45:28.461699    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e\": container with ID starting with cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e not found: ID does not exist" containerID="cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"
May 26 11:45:28.461845 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.461732    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e"} err="failed to get container status \"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e\": rpc error: code = NotFound desc = could not find container \"cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e\": container with ID starting with cc692575b92a2d83303fc89ab47763753b278fccc1e2f8a7d541ecd586e12b1e not found: ID does not exist"
May 26 11:45:28.461845 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.461769    2562 scope.go:117] "RemoveContainer" containerID="9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f"
May 26 11:45:28.462361 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:45:28.462329    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f\": container with ID starting with 9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f not found: ID does not exist" containerID="9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f"
May 26 11:45:28.462470 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.462358    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f"} err="failed to get container status \"9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f\": rpc error: code = NotFound desc = could not find container \"9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f\": container with ID starting with 9257d03e5146ce4d4e5e0db777a593a9f43c76e4fb68c2c978508cd85c1a918f not found: ID does not exist"
May 26 11:45:28.463818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.463800    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7474554jp8td"]
May 26 11:45:28.679186 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:28.679154    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" path="/var/lib/kubelet/pods/b156354d-cbd4-4366-9e6d-5828959a86a6/volumes"
May 26 11:45:33.648113 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648081    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648392    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="main"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648403    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="main"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648427    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="tokenizer"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648435    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="tokenizer"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648457    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="storage-initializer"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648463    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="storage-initializer"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648509    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="tokenizer"
May 26 11:45:33.648551 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.648520    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="b156354d-cbd4-4366-9e6d-5828959a86a6" containerName="main"
May 26 11:45:33.653278 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.653262    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.655782 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.655759    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 26 11:45:33.661518 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.661493    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 11:45:33.750355 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.750564 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.750544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd68k\" (UniqueName: \"kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851675 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851638    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851675 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851672    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851705    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851723    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pd68k\" (UniqueName: \"kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851767    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.851876 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851804    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.852056 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.851963    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.852122 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.852103    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.852179 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.852121    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.852235 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.852174    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.852352 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.852334    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.853994 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.853977    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.854235 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.854219    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.859423 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.859393    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd68k\" (UniqueName: \"kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k\") pod \"precise-prefix-cache-test-kserve-84f494576-7cbvm\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:33.964222 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:33.964141    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:45:34.088937 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:34.088905    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 11:45:34.092027 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:45:34.091999    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8104ce1e_b9d5_4853_8b92_e1f141514193.slice/crio-3dab829c7eb860212b8449083bfe3b671408d31ce0f4c044f9cade501248a767 WatchSource:0}: Error finding container 3dab829c7eb860212b8449083bfe3b671408d31ce0f4c044f9cade501248a767: Status 404 returned error can't find the container with id 3dab829c7eb860212b8449083bfe3b671408d31ce0f4c044f9cade501248a767
May 26 11:45:34.459966 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:34.459928    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerStarted","Data":"fe5caab40e54348d34fab9a7254a4b2e080fff2455fd54d19f2e99dfa8793a44"}
May 26 11:45:34.459966 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:45:34.459964    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerStarted","Data":"3dab829c7eb860212b8449083bfe3b671408d31ce0f4c044f9cade501248a767"}
May 26 11:46:25.636913 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:25.636877    2562 generic.go:358] "Generic (PLEG): container finished" podID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerID="fe5caab40e54348d34fab9a7254a4b2e080fff2455fd54d19f2e99dfa8793a44" exitCode=0
May 26 11:46:25.637377 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:25.636941    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerDied","Data":"fe5caab40e54348d34fab9a7254a4b2e080fff2455fd54d19f2e99dfa8793a44"}
May 26 11:46:26.641325 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:26.641294    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerStarted","Data":"6a1e101c9e572a785221ef3bed94ff23a5648c0e774ee24c168e431809319c4c"}
May 26 11:46:26.669905 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:26.669859    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" podStartSLOduration=53.669844671999996 podStartE2EDuration="53.669844672s" podCreationTimestamp="2026-05-26 11:45:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:46:26.666168245 +0000 UTC m=+1282.626715732" watchObservedRunningTime="2026-05-26 11:46:26.669844672 +0000 UTC m=+1282.630392177"
May 26 11:46:33.964955 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:33.964902    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:46:33.964955 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:33.964964    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:46:33.977369 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:33.977340    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:46:34.680386 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:34.680360    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 11:46:58.533019 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:58.532980    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:46:58.533563 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:58.533366    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="main" containerID="cri-o://58628cdaf7ca857eaf56ac950687e595b000f0468f342d76e436daf51b906dfd" gracePeriod=30
May 26 11:46:58.533563 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:58.533465    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="tokenizer" containerID="cri-o://42055e5400d6e4b6cfdf51eae24344752806707ad711b226fc0d82446009214f" gracePeriod=30
May 26 11:46:58.743979 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:58.743943    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerID="58628cdaf7ca857eaf56ac950687e595b000f0468f342d76e436daf51b906dfd" exitCode=0
May 26 11:46:58.744178 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:58.744014    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerDied","Data":"58628cdaf7ca857eaf56ac950687e595b000f0468f342d76e436daf51b906dfd"}
May 26 11:46:59.748796 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.748762    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerID="42055e5400d6e4b6cfdf51eae24344752806707ad711b226fc0d82446009214f" exitCode=0
May 26 11:46:59.749142 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.748831    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerDied","Data":"42055e5400d6e4b6cfdf51eae24344752806707ad711b226fc0d82446009214f"}
May 26 11:46:59.780314 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.780290    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:46:59.894378 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894351    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894393    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894454    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894494    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894558 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894540    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894765 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894574    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x58lm\" (UniqueName: \"kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm\") pod \"ca24a014-8fc3-49e8-bbd1-480466f10c83\" (UID: \"ca24a014-8fc3-49e8-bbd1-480466f10c83\") "
May 26 11:46:59.894765 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894715    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:59.894765 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894751    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:59.894912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894875    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:59.894912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894876    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:46:59.894912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.894912    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:46:59.895151 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.895132    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:46:59.896676 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.896644    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:46:59.896676 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.896659    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm" (OuterVolumeSpecName: "kube-api-access-x58lm") pod "ca24a014-8fc3-49e8-bbd1-480466f10c83" (UID: "ca24a014-8fc3-49e8-bbd1-480466f10c83"). InnerVolumeSpecName "kube-api-access-x58lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:46:59.995880 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.995839    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca24a014-8fc3-49e8-bbd1-480466f10c83-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:46:59.995880 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.995872    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58lm\" (UniqueName: \"kubernetes.io/projected/ca24a014-8fc3-49e8-bbd1-480466f10c83-kube-api-access-x58lm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:46:59.995880 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.995884    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:46:59.996103 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:46:59.995895    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca24a014-8fc3-49e8-bbd1-480466f10c83-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:47:00.753865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.753833    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9" event={"ID":"ca24a014-8fc3-49e8-bbd1-480466f10c83","Type":"ContainerDied","Data":"506c42a73a9a9bc5115c0708cb96e8fa2a8d05ab6aa5cbdc1cfd4f81985fdf79"}
May 26 11:47:00.753865 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.753857    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"
May 26 11:47:00.754349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.753882    2562 scope.go:117] "RemoveContainer" containerID="42055e5400d6e4b6cfdf51eae24344752806707ad711b226fc0d82446009214f"
May 26 11:47:00.762320 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.762303    2562 scope.go:117] "RemoveContainer" containerID="58628cdaf7ca857eaf56ac950687e595b000f0468f342d76e436daf51b906dfd"
May 26 11:47:00.769844 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.769824    2562 scope.go:117] "RemoveContainer" containerID="eee32644c7bd7dd3d8ece116dc669090fb961c5aba3b711a0c75c126ad673dfe"
May 26 11:47:00.774548 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.774525    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:47:00.778903 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:00.778869    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2qlx9"]
May 26 11:47:02.679265 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:02.679231    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" path="/var/lib/kubelet/pods/ca24a014-8fc3-49e8-bbd1-480466f10c83/volumes"
May 26 11:47:12.402692 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.402658    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.402999    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="storage-initializer"
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403012    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="storage-initializer"
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403035    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="tokenizer"
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403042    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="tokenizer"
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403049    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="main"
May 26 11:47:12.403079 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="main"
May 26 11:47:12.403329 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403128    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="main"
May 26 11:47:12.403329 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.403139    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca24a014-8fc3-49e8-bbd1-480466f10c83" containerName="tokenizer"
May 26 11:47:12.408022 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.408001    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.410887 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.410867    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 26 11:47:12.411140 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.411124    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-2wc6f\""
May 26 11:47:12.418739 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.418714    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:47:12.504700 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzshn\" (UniqueName: \"kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.504842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.504842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.504842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.504842 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.504990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.504844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606241 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606203    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606241 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606246    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606477 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606300    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606477 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606408    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606543 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606475    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606543 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606529    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gzshn\" (UniqueName: \"kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606700 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606678    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606767 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606714    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606825 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606805    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.606874 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.606822    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.608818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.608796    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.617404 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.617382    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzshn\" (UniqueName: \"kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn\") pod \"custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.718623 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.718538    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:12.870586 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:12.870107    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:47:12.873097 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:47:12.873066    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa22da47_e04c_451b_b5aa_5eea1816a03a.slice/crio-337aad30e9ecdc9c4865696eb5d6d2905a86984cf9d6846891b624c9e86a2822 WatchSource:0}: Error finding container 337aad30e9ecdc9c4865696eb5d6d2905a86984cf9d6846891b624c9e86a2822: Status 404 returned error can't find the container with id 337aad30e9ecdc9c4865696eb5d6d2905a86984cf9d6846891b624c9e86a2822
May 26 11:47:13.802836 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:13.802802    2562 generic.go:358] "Generic (PLEG): container finished" podID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerID="62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63" exitCode=0
May 26 11:47:13.803204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:13.802888    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerDied","Data":"62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63"}
May 26 11:47:13.803204 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:13.802923    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerStarted","Data":"337aad30e9ecdc9c4865696eb5d6d2905a86984cf9d6846891b624c9e86a2822"}
May 26 11:47:14.808490 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:14.808449    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerStarted","Data":"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"}
May 26 11:47:14.808490 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:14.808495    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerStarted","Data":"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"}
May 26 11:47:14.808925 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:14.808759    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:14.830552 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:14.830503    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" podStartSLOduration=2.830486859 podStartE2EDuration="2.830486859s" podCreationTimestamp="2026-05-26 11:47:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:47:14.828709799 +0000 UTC m=+1330.789257313" watchObservedRunningTime="2026-05-26 11:47:14.830486859 +0000 UTC m=+1330.791034359"
May 26 11:47:22.718944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:22.718857    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:22.718944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:22.718907    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:22.721578 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:22.721556    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:22.838981 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:22.838949    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:47:43.843611 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:47:43.843580    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:50:04.661293 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:04.661262    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:50:04.662486 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:04.662464    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:50:38.686601 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:38.686503    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:50:38.687183 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:38.686967    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="main" containerID="cri-o://e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75" gracePeriod=30
May 26 11:50:38.687183 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:38.687095    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="tokenizer" containerID="cri-o://b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011" gracePeriod=30
May 26 11:50:39.503997 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:39.503959    2562 generic.go:358] "Generic (PLEG): container finished" podID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerID="e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75" exitCode=0
May 26 11:50:39.504195 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:39.504034    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerDied","Data":"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"}
May 26 11:50:39.932967 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:39.932944    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:50:40.015519 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015489    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015528    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzshn\" (UniqueName: \"kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015546    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015572    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015632    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015823 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015668    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache\") pod \"aa22da47-e04c-451b-b5aa-5eea1816a03a\" (UID: \"aa22da47-e04c-451b-b5aa-5eea1816a03a\") "
May 26 11:50:40.015862 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015826    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:50:40.015920 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015890    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.015976 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015936    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:50:40.016030 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.015991    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:50:40.016291 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.016270    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:50:40.017659 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.017637    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn" (OuterVolumeSpecName: "kube-api-access-gzshn") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "kube-api-access-gzshn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:50:40.017716 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.017659    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa22da47-e04c-451b-b5aa-5eea1816a03a" (UID: "aa22da47-e04c-451b-b5aa-5eea1816a03a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:50:40.116830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.116734    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.116830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.116780    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.116830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.116791    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzshn\" (UniqueName: \"kubernetes.io/projected/aa22da47-e04c-451b-b5aa-5eea1816a03a-kube-api-access-gzshn\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.116830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.116800    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa22da47-e04c-451b-b5aa-5eea1816a03a-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.116830 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.116811    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa22da47-e04c-451b-b5aa-5eea1816a03a-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:50:40.508978 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.508941    2562 generic.go:358] "Generic (PLEG): container finished" podID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerID="b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011" exitCode=0
May 26 11:50:40.509163 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.508998    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerDied","Data":"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"}
May 26 11:50:40.509163 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.509025    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"
May 26 11:50:40.509163 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.509035    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g" event={"ID":"aa22da47-e04c-451b-b5aa-5eea1816a03a","Type":"ContainerDied","Data":"337aad30e9ecdc9c4865696eb5d6d2905a86984cf9d6846891b624c9e86a2822"}
May 26 11:50:40.509163 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.509053    2562 scope.go:117] "RemoveContainer" containerID="b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"
May 26 11:50:40.517199 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.517180    2562 scope.go:117] "RemoveContainer" containerID="e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"
May 26 11:50:40.524364 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.524343    2562 scope.go:117] "RemoveContainer" containerID="62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63"
May 26 11:50:40.531472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.531451    2562 scope.go:117] "RemoveContainer" containerID="b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"
May 26 11:50:40.531728 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:50:40.531703    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011\": container with ID starting with b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011 not found: ID does not exist" containerID="b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"
May 26 11:50:40.531812 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.531731    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011"} err="failed to get container status \"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011\": rpc error: code = NotFound desc = could not find container \"b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011\": container with ID starting with b81d92819a169240d5124726ee2fb8b24971d18f6fb2dd5d51a2029c7a2fd011 not found: ID does not exist"
May 26 11:50:40.531812 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.531751    2562 scope.go:117] "RemoveContainer" containerID="e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"
May 26 11:50:40.531956 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:50:40.531939    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75\": container with ID starting with e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75 not found: ID does not exist" containerID="e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"
May 26 11:50:40.532000 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.531959    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75"} err="failed to get container status \"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75\": rpc error: code = NotFound desc = could not find container \"e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75\": container with ID starting with e748680428ae8d1609ded7434d505283fd2033a21b81bdfeda2706de8c859d75 not found: ID does not exist"
May 26 11:50:40.532000 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.531972    2562 scope.go:117] "RemoveContainer" containerID="62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63"
May 26 11:50:40.532225 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:50:40.532202    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63\": container with ID starting with 62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63 not found: ID does not exist" containerID="62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63"
May 26 11:50:40.532283 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.532233    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63"} err="failed to get container status \"62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63\": rpc error: code = NotFound desc = could not find container \"62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63\": container with ID starting with 62145e5acbcf82168803d5bcf1a519b24b84224d4315bab1cedcb4e988de7d63 not found: ID does not exist"
May 26 11:50:40.547635 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.547610    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:50:40.562349 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.562319    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5798769bmpr9g"]
May 26 11:50:40.679364 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:40.679333    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" path="/var/lib/kubelet/pods/aa22da47-e04c-451b-b5aa-5eea1816a03a/volumes"
May 26 11:50:47.799127 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799091    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799439    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="tokenizer"
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799454    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="tokenizer"
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799471    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="storage-initializer"
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799477    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="storage-initializer"
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799497    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="main"
May 26 11:50:47.799527 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799502    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="main"
May 26 11:50:47.799728 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799550    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="main"
May 26 11:50:47.799728 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.799562    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="aa22da47-e04c-451b-b5aa-5eea1816a03a" containerName="tokenizer"
May 26 11:50:47.803883 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.803862    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.807033 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.807016    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-8ctbw\""
May 26 11:50:47.807672 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.807644    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 26 11:50:47.815472 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.815450    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:50:47.883383 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.883383 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.883621 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.883621 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ngm\" (UniqueName: \"kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.883621 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.883621 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.883592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985026 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.984993    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ngm\" (UniqueName: \"kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985026 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985033    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985067    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985104    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985130    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985148    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985561 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985536    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985607 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985586    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985668 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.985866 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.985841    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:47.987773 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:47.987748    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:48.002347 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.002321    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ngm\" (UniqueName: \"kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm\") pod \"router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:48.118829 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.118752    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:48.254178 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.253982    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:50:48.257159 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:50:48.257134    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6907f8ae_f28a_4a28_854e_47ad838f5049.slice/crio-fa54552fdac09441d61bd247d152064063f103d9d982287cad80e3e1d1a01358 WatchSource:0}: Error finding container fa54552fdac09441d61bd247d152064063f103d9d982287cad80e3e1d1a01358: Status 404 returned error can't find the container with id fa54552fdac09441d61bd247d152064063f103d9d982287cad80e3e1d1a01358
May 26 11:50:48.259406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.259390    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:50:48.536482 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.536447    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerStarted","Data":"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"}
May 26 11:50:48.536482 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:48.536487    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerStarted","Data":"fa54552fdac09441d61bd247d152064063f103d9d982287cad80e3e1d1a01358"}
May 26 11:50:49.541206 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:49.541164    2562 generic.go:358] "Generic (PLEG): container finished" podID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerID="eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140" exitCode=0
May 26 11:50:49.541616 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:49.541246    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerDied","Data":"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"}
May 26 11:50:50.547047 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:50.547010    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerStarted","Data":"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"}
May 26 11:50:50.547047 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:50.547047    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerStarted","Data":"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"}
May 26 11:50:50.547572 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:50.547129    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:50.571283 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:50.571226    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" podStartSLOduration=3.571205853 podStartE2EDuration="3.571205853s" podCreationTimestamp="2026-05-26 11:50:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:50:50.569046019 +0000 UTC m=+1546.529593514" watchObservedRunningTime="2026-05-26 11:50:50.571205853 +0000 UTC m=+1546.531753340"
May 26 11:50:58.119131 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:58.119093    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:58.119565 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:58.119149    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:58.121743 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:58.121721    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:50:58.575325 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:50:58.575297    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:51:19.579342 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:51:19.579311    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:52:34.124268 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:34.124230    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:52:34.124789 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:34.124676    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="main" containerID="cri-o://dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe" gracePeriod=30
May 26 11:52:34.124789 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:34.124724    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="tokenizer" containerID="cri-o://d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec" gracePeriod=30
May 26 11:52:34.894033 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:34.894000    2562 generic.go:358] "Generic (PLEG): container finished" podID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerID="dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe" exitCode=0
May 26 11:52:34.894189 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:34.894048    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerDied","Data":"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"}
May 26 11:52:35.375143 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.375118    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:52:35.406314 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406245    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406314 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406288    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406520 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406343    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6ngm\" (UniqueName: \"kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406520 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406390    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406520 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406485    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406664 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406536    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs\") pod \"6907f8ae-f28a-4a28-854e-47ad838f5049\" (UID: \"6907f8ae-f28a-4a28-854e-47ad838f5049\") "
May 26 11:52:35.406664 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406559    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:52:35.406769 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406757    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.406822 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406774    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:52:35.406822 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.406788    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:52:35.407083 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.407062    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:52:35.408742 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.408716    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:52:35.408742 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.408726    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm" (OuterVolumeSpecName: "kube-api-access-s6ngm") pod "6907f8ae-f28a-4a28-854e-47ad838f5049" (UID: "6907f8ae-f28a-4a28-854e-47ad838f5049"). InnerVolumeSpecName "kube-api-access-s6ngm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:52:35.507660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.507626    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.507660 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.507659    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6907f8ae-f28a-4a28-854e-47ad838f5049-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.507860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.507674    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.507860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.507687    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6ngm\" (UniqueName: \"kubernetes.io/projected/6907f8ae-f28a-4a28-854e-47ad838f5049-kube-api-access-s6ngm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.507860 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.507696    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6907f8ae-f28a-4a28-854e-47ad838f5049-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:52:35.899170 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.899141    2562 generic.go:358] "Generic (PLEG): container finished" podID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerID="d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec" exitCode=0
May 26 11:52:35.899339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.899190    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerDied","Data":"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"}
May 26 11:52:35.899339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.899210    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj" event={"ID":"6907f8ae-f28a-4a28-854e-47ad838f5049","Type":"ContainerDied","Data":"fa54552fdac09441d61bd247d152064063f103d9d982287cad80e3e1d1a01358"}
May 26 11:52:35.899339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.899212    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"
May 26 11:52:35.899339 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.899227    2562 scope.go:117] "RemoveContainer" containerID="d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"
May 26 11:52:35.907893 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.907873    2562 scope.go:117] "RemoveContainer" containerID="dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"
May 26 11:52:35.917517 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.917356    2562 scope.go:117] "RemoveContainer" containerID="eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"
May 26 11:52:35.921450 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.921409    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:52:35.926061 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.926036    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-74dcd655dc-zngbj"]
May 26 11:52:35.927005 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.926981    2562 scope.go:117] "RemoveContainer" containerID="d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"
May 26 11:52:35.927304 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:52:35.927286    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec\": container with ID starting with d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec not found: ID does not exist" containerID="d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"
May 26 11:52:35.927387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.927313    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec"} err="failed to get container status \"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec\": rpc error: code = NotFound desc = could not find container \"d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec\": container with ID starting with d05f111b1f5e69a9d6df7989c258aff432cc66161430fa8a4d13708257d80bec not found: ID does not exist"
May 26 11:52:35.927387 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.927330    2562 scope.go:117] "RemoveContainer" containerID="dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"
May 26 11:52:35.927624 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:52:35.927604    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe\": container with ID starting with dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe not found: ID does not exist" containerID="dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"
May 26 11:52:35.927666 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.927638    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe"} err="failed to get container status \"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe\": rpc error: code = NotFound desc = could not find container \"dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe\": container with ID starting with dfc041158f1ccf51e0b335a158e90971e7a0dac8b2341c09bb6601aeb20617fe not found: ID does not exist"
May 26 11:52:35.927666 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.927654    2562 scope.go:117] "RemoveContainer" containerID="eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"
May 26 11:52:35.927861 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:52:35.927848    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140\": container with ID starting with eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140 not found: ID does not exist" containerID="eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"
May 26 11:52:35.927912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:35.927863    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140"} err="failed to get container status \"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140\": rpc error: code = NotFound desc = could not find container \"eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140\": container with ID starting with eb8a864dc04de85ed49e4fb30c043f5a855ca5e6bb5d0e1a850b61b3cd39e140 not found: ID does not exist"
May 26 11:52:36.679671 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:36.679629    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" path="/var/lib/kubelet/pods/6907f8ae-f28a-4a28-854e-47ad838f5049/volumes"
May 26 11:52:44.698105 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698071    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698398    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="tokenizer"
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698410    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="tokenizer"
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698436    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="main"
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698445    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="main"
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698465    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="storage-initializer"
May 26 11:52:44.698476 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698474    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="storage-initializer"
May 26 11:52:44.698663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698568    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="main"
May 26 11:52:44.698663 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.698580    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="6907f8ae-f28a-4a28-854e-47ad838f5049" containerName="tokenizer"
May 26 11:52:44.703483 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.703462    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.706687 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.706664    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 26 11:52:44.706789 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.706674    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-r4vdx\""
May 26 11:52:44.717228 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.717205    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:52:44.785286 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.785507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.785507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.785507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvck\" (UniqueName: \"kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.785507 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.785698 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.785566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886344 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886309    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886350    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886514 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886391    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886588 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886551    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886635 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886617    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvck\" (UniqueName: \"kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886690 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886673    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886819 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886797    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886902 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886825    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886956 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.886956 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.886929    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.888622 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.888600    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:44.897234 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:44.897208    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvck\" (UniqueName: \"kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:45.012173 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:45.012093    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:45.141753 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:45.141714    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:52:45.144007 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:52:45.143982    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7fd36a_7663_4656_b36a_814ff11d4e85.slice/crio-02cde2c75138f7456a9b1065380cc9795e78f241406e7420ab9e16a03a7d2fdc WatchSource:0}: Error finding container 02cde2c75138f7456a9b1065380cc9795e78f241406e7420ab9e16a03a7d2fdc: Status 404 returned error can't find the container with id 02cde2c75138f7456a9b1065380cc9795e78f241406e7420ab9e16a03a7d2fdc
May 26 11:52:45.934406 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:45.934368    2562 generic.go:358] "Generic (PLEG): container finished" podID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerID="82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c" exitCode=0
May 26 11:52:45.934761 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:45.934467    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerDied","Data":"82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c"}
May 26 11:52:45.934761 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:45.934513    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerStarted","Data":"02cde2c75138f7456a9b1065380cc9795e78f241406e7420ab9e16a03a7d2fdc"}
May 26 11:52:46.939840 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:46.939804    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerStarted","Data":"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"}
May 26 11:52:46.939840 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:46.939841    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerStarted","Data":"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"}
May 26 11:52:46.940240 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:46.939997    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:46.964484 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:46.964407    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" podStartSLOduration=2.964387182 podStartE2EDuration="2.964387182s" podCreationTimestamp="2026-05-26 11:52:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:52:46.961861039 +0000 UTC m=+1662.922408528" watchObservedRunningTime="2026-05-26 11:52:46.964387182 +0000 UTC m=+1662.924934669"
May 26 11:52:55.012522 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:55.012486    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:55.012912 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:55.012636    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:55.015267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:55.015244    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:52:55.970841 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:52:55.970807    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:53:17.978209 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:53:17.978129    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:55:04.693562 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:04.693535    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:55:04.695533 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:04.695507    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 11:55:12.992818 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:12.992781    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:55:12.993242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:12.993095    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="main" containerID="cri-o://b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31" gracePeriod=30
May 26 11:55:12.993242 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:12.993150    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="tokenizer" containerID="cri-o://107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb" gracePeriod=30
May 26 11:55:13.417995 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:13.417959    2562 generic.go:358] "Generic (PLEG): container finished" podID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerID="b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31" exitCode=0
May 26 11:55:13.418161 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:13.418030    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerDied","Data":"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"}
May 26 11:55:14.233884 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.233863    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:55:14.376035 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376008    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kvck\" (UniqueName: \"kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376193 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376066    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376193 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376093    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376328 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376200    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376328 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376231    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376328 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376275    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location\") pod \"ea7fd36a-7663-4656-b36a-814ff11d4e85\" (UID: \"ea7fd36a-7663-4656-b36a-814ff11d4e85\") "
May 26 11:55:14.376519 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376340    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:14.376519 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376466    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:14.376519 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376490    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:14.376665 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376640    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.376665 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376659    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.376742 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376673    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.377003 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.376983    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:14.378166 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.378134    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck" (OuterVolumeSpecName: "kube-api-access-6kvck") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "kube-api-access-6kvck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:14.378267 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.378254    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea7fd36a-7663-4656-b36a-814ff11d4e85" (UID: "ea7fd36a-7663-4656-b36a-814ff11d4e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:14.422385 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.422345    2562 generic.go:358] "Generic (PLEG): container finished" podID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerID="107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb" exitCode=0
May 26 11:55:14.422571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.422438    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"
May 26 11:55:14.422571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.422452    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerDied","Data":"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"}
May 26 11:55:14.422571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.422501    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht" event={"ID":"ea7fd36a-7663-4656-b36a-814ff11d4e85","Type":"ContainerDied","Data":"02cde2c75138f7456a9b1065380cc9795e78f241406e7420ab9e16a03a7d2fdc"}
May 26 11:55:14.422571 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.422539    2562 scope.go:117] "RemoveContainer" containerID="107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"
May 26 11:55:14.430597 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.430545    2562 scope.go:117] "RemoveContainer" containerID="b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"
May 26 11:55:14.437199 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.437185    2562 scope.go:117] "RemoveContainer" containerID="82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c"
May 26 11:55:14.443401 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.443380    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:55:14.445009 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.444993    2562 scope.go:117] "RemoveContainer" containerID="107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"
May 26 11:55:14.445288 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:55:14.445269    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb\": container with ID starting with 107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb not found: ID does not exist" containerID="107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"
May 26 11:55:14.445336 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.445300    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb"} err="failed to get container status \"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb\": rpc error: code = NotFound desc = could not find container \"107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb\": container with ID starting with 107ab6dcd9f49760a85629ccae10b2b6f2f6f43fb11f4920fbdb06d8aaf8f0cb not found: ID does not exist"
May 26 11:55:14.445336 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.445322    2562 scope.go:117] "RemoveContainer" containerID="b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"
May 26 11:55:14.445594 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:55:14.445569    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31\": container with ID starting with b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31 not found: ID does not exist" containerID="b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"
May 26 11:55:14.445653 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.445605    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31"} err="failed to get container status \"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31\": rpc error: code = NotFound desc = could not find container \"b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31\": container with ID starting with b455e3b162b75f3d7f39113081a5fd85ee2199cc7d8a7a33c964e59c818fbb31 not found: ID does not exist"
May 26 11:55:14.445653 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.445642    2562 scope.go:117] "RemoveContainer" containerID="82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c"
May 26 11:55:14.446138 ip-10-0-133-125 kubenswrapper[2562]: E0526 11:55:14.446110    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c\": container with ID starting with 82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c not found: ID does not exist" containerID="82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c"
May 26 11:55:14.446259 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.446144    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c"} err="failed to get container status \"82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c\": rpc error: code = NotFound desc = could not find container \"82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c\": container with ID starting with 82d2ae234ca13c96d5f624e77628d9422ea5e7ec19f0aad21e8d54307c511a7c not found: ID does not exist"
May 26 11:55:14.447955 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.447933    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sche25jht"]
May 26 11:55:14.477613 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.477573    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7fd36a-7663-4656-b36a-814ff11d4e85-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.477613 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.477611    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea7fd36a-7663-4656-b36a-814ff11d4e85-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.477613 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.477622    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvck\" (UniqueName: \"kubernetes.io/projected/ea7fd36a-7663-4656-b36a-814ff11d4e85-kube-api-access-6kvck\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 11:55:14.679761 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:14.679682    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" path="/var/lib/kubelet/pods/ea7fd36a-7663-4656-b36a-814ff11d4e85/volumes"
May 26 11:55:22.474907 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.474869    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475190    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="tokenizer"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475200    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="tokenizer"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475215    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="storage-initializer"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="storage-initializer"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475227    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="main"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475232    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="main"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475290    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="main"
May 26 11:55:22.475333 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.475296    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea7fd36a-7663-4656-b36a-814ff11d4e85" containerName="tokenizer"
May 26 11:55:22.479909 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.479887    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.482381 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.482349    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-mt6dw\""
May 26 11:55:22.482509 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.482456    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 26 11:55:22.490361 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.490341    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 11:55:22.648863 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.648824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.648863 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.648870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.649072 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.648927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqh5l\" (UniqueName: \"kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.649072 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.648970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.649072 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.649013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.649072 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.649055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.749990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.749899    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.749990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.749942    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.749990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.749966    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gqh5l\" (UniqueName: \"kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.749990 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.749991    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750019    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750317 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750059    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750465 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750356    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750465 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750406    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750502    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.750550 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.750528    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.752624 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.752600    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.758646 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.758622    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqh5l\" (UniqueName: \"kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.788544 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.788509    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:22.921287 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:22.921254    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 11:55:22.924522 ip-10-0-133-125 kubenswrapper[2562]: W0526 11:55:22.924490    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca590ad5_2ac8_4f88_a5f8_9956198b0601.slice/crio-344fe568177de4dfb0d92d4a5ce2833b1bbde1cb59e2a314126cddc6c0db76eb WatchSource:0}: Error finding container 344fe568177de4dfb0d92d4a5ce2833b1bbde1cb59e2a314126cddc6c0db76eb: Status 404 returned error can't find the container with id 344fe568177de4dfb0d92d4a5ce2833b1bbde1cb59e2a314126cddc6c0db76eb
May 26 11:55:23.454703 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:23.454669    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerStarted","Data":"9ddd7e42b8ee621f61df746f0d1ddd12b18da34d4046a603eecb818541ea8774"}
May 26 11:55:23.454703 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:23.454711    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerStarted","Data":"344fe568177de4dfb0d92d4a5ce2833b1bbde1cb59e2a314126cddc6c0db76eb"}
May 26 11:55:24.458969 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:24.458935    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerID="9ddd7e42b8ee621f61df746f0d1ddd12b18da34d4046a603eecb818541ea8774" exitCode=0
May 26 11:55:24.459433 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:24.459021    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerDied","Data":"9ddd7e42b8ee621f61df746f0d1ddd12b18da34d4046a603eecb818541ea8774"}
May 26 11:55:25.464061 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:25.464025    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerStarted","Data":"08a84d1e435d546bd42bfe2394aaac38b6e40fd232c99f3dabb3a52a1733c8a7"}
May 26 11:55:25.464061 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:25.464065    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerStarted","Data":"a0489d5b50811a2f22d4c9bc18bf18953f33e235b28f94910ff019b09dec924f"}
May 26 11:55:25.464547 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:25.464126    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:25.486229 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:25.486162    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" podStartSLOduration=3.486142789 podStartE2EDuration="3.486142789s" podCreationTimestamp="2026-05-26 11:55:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:25.483716861 +0000 UTC m=+1821.444264371" watchObservedRunningTime="2026-05-26 11:55:25.486142789 +0000 UTC m=+1821.446690275"
May 26 11:55:32.789666 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:32.789631    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:32.789666 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:32.789664    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:32.792270 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:32.792243    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:33.492944 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:33.492915    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 11:55:54.498067 ip-10-0-133-125 kubenswrapper[2562]: I0526 11:55:54.498038    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 12:00:04.722495 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:04.722469    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:00:04.725225 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:04.725200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:00:29.289777 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.289737    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 12:00:29.290234 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.290077    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="main" containerID="cri-o://6a1e101c9e572a785221ef3bed94ff23a5648c0e774ee24c168e431809319c4c" gracePeriod=30
May 26 12:00:29.471978 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.471941    2562 generic.go:358] "Generic (PLEG): container finished" podID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerID="6a1e101c9e572a785221ef3bed94ff23a5648c0e774ee24c168e431809319c4c" exitCode=0
May 26 12:00:29.472111 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.472009    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerDied","Data":"6a1e101c9e572a785221ef3bed94ff23a5648c0e774ee24c168e431809319c4c"}
May 26 12:00:29.531111 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.531086    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 12:00:29.705405 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705372    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705405 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705411    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705679 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705451    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705679 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705469    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705679 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705498    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705679 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705584    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705679 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705622    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd68k\" (UniqueName: \"kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k\") pod \"8104ce1e-b9d5-4853-8b92-e1f141514193\" (UID: \"8104ce1e-b9d5-4853-8b92-e1f141514193\") "
May 26 12:00:29.705948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705784    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:29.705948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705917    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home" (OuterVolumeSpecName: "home") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:29.705948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.705916    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache" (OuterVolumeSpecName: "model-cache") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:29.707671 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.707645    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:00:29.707781 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.707701    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k" (OuterVolumeSpecName: "kube-api-access-pd68k") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "kube-api-access-pd68k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:00:29.707905 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.707889    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm" (OuterVolumeSpecName: "dshm") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:29.760613 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.760564    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8104ce1e-b9d5-4853-8b92-e1f141514193" (UID: "8104ce1e-b9d5-4853-8b92-e1f141514193"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:29.806694 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806660    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806694 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806689    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd68k\" (UniqueName: \"kubernetes.io/projected/8104ce1e-b9d5-4853-8b92-e1f141514193-kube-api-access-pd68k\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806694 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806700    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806902 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806710    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806902 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806720    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806902 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806728    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8104ce1e-b9d5-4853-8b92-e1f141514193-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:29.806902 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:29.806736    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8104ce1e-b9d5-4853-8b92-e1f141514193-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:00:30.476244 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.476217    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"
May 26 12:00:30.476729 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.476220    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm" event={"ID":"8104ce1e-b9d5-4853-8b92-e1f141514193","Type":"ContainerDied","Data":"3dab829c7eb860212b8449083bfe3b671408d31ce0f4c044f9cade501248a767"}
May 26 12:00:30.476729 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.476343    2562 scope.go:117] "RemoveContainer" containerID="6a1e101c9e572a785221ef3bed94ff23a5648c0e774ee24c168e431809319c4c"
May 26 12:00:30.485325 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.485273    2562 scope.go:117] "RemoveContainer" containerID="fe5caab40e54348d34fab9a7254a4b2e080fff2455fd54d19f2e99dfa8793a44"
May 26 12:00:30.499037 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.498990    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 12:00:30.501740 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.501719    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-7cbvm"]
May 26 12:00:30.679098 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:30.679069    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" path="/var/lib/kubelet/pods/8104ce1e-b9d5-4853-8b92-e1f141514193/volumes"
May 26 12:00:39.676821 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.676788    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:00:39.677314 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.677257    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="storage-initializer"
May 26 12:00:39.677314 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.677276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="storage-initializer"
May 26 12:00:39.677314 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.677291    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="main"
May 26 12:00:39.677314 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.677299    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="main"
May 26 12:00:39.677567 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.677376    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="8104ce1e-b9d5-4853-8b92-e1f141514193" containerName="main"
May 26 12:00:39.682550 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.682529    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.686492 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.686470    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-q2ljh\""
May 26 12:00:39.686607 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.686580    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 12:00:39.690337 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.690312    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:00:39.794052 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.794251 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.794251 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g746n\" (UniqueName: \"kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.794400 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.794400 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.794515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.794407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895080 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895047    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895237 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895102    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895237 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895121    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g746n\" (UniqueName: \"kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895237 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895162    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895237 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895182    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895637 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895309    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895637 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895517    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895637 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895537    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895637 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895568    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.895816 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.895674    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.897517 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.897495    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.903068 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.903041    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g746n\" (UniqueName: \"kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:39.992650 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:39.992571    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:40.126203 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:40.126158    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:00:40.130061 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:00:40.130031    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722df366_af36_4769_9fb5_8fd5cfd815f1.slice/crio-aa784cc9a4dc7130e12bcf9fb5bc0f908c8229862cfaa78c6a1a4cb62b5055b2 WatchSource:0}: Error finding container aa784cc9a4dc7130e12bcf9fb5bc0f908c8229862cfaa78c6a1a4cb62b5055b2: Status 404 returned error can't find the container with id aa784cc9a4dc7130e12bcf9fb5bc0f908c8229862cfaa78c6a1a4cb62b5055b2
May 26 12:00:40.132482 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:40.132464    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:00:40.511910 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:40.511869    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerStarted","Data":"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"}
May 26 12:00:40.512102 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:40.511919    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerStarted","Data":"aa784cc9a4dc7130e12bcf9fb5bc0f908c8229862cfaa78c6a1a4cb62b5055b2"}
May 26 12:00:41.516157 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:41.516118    2562 generic.go:358] "Generic (PLEG): container finished" podID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerID="8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517" exitCode=0
May 26 12:00:41.516605 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:41.516200    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerDied","Data":"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"}
May 26 12:00:42.522207 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:42.522174    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerStarted","Data":"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"}
May 26 12:00:42.522207 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:42.522208    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerStarted","Data":"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"}
May 26 12:00:42.522644 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:42.522443    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:42.544349 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:42.544299    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" podStartSLOduration=3.544283268 podStartE2EDuration="3.544283268s" podCreationTimestamp="2026-05-26 12:00:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:00:42.54199611 +0000 UTC m=+2138.502543596" watchObservedRunningTime="2026-05-26 12:00:42.544283268 +0000 UTC m=+2138.504830753"
May 26 12:00:49.993180 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:49.993095    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:49.993180 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:49.993146    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:49.996007 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:49.995981    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:00:50.553539 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:00:50.553511    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:01:11.558483 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:01:11.558454    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:04:40.977253 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:40.977219    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:04:40.977801 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:40.977542    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="main" containerID="cri-o://2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a" gracePeriod=30
May 26 12:04:40.977801 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:40.977596    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="tokenizer" containerID="cri-o://07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623" gracePeriod=30
May 26 12:04:41.327130 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:41.327040    2562 generic.go:358] "Generic (PLEG): container finished" podID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerID="2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a" exitCode=0
May 26 12:04:41.327130 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:41.327109    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerDied","Data":"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"}
May 26 12:04:41.557059 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:04:41.557028    2562 logging.go:55] [core] [Channel #968 SubChannel #969]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.35:9003", ServerName: "10.134.0.35:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.35:9003: connect: connection refused"
May 26 12:04:42.231743 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.231719    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:04:42.337515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.337396    2562 generic.go:358] "Generic (PLEG): container finished" podID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerID="07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623" exitCode=0
May 26 12:04:42.337515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.337446    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerDied","Data":"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"}
May 26 12:04:42.337515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.337489    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" event={"ID":"722df366-af36-4769-9fb5-8fd5cfd815f1","Type":"ContainerDied","Data":"aa784cc9a4dc7130e12bcf9fb5bc0f908c8229862cfaa78c6a1a4cb62b5055b2"}
May 26 12:04:42.337515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.337505    2562 scope.go:117] "RemoveContainer" containerID="07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"
May 26 12:04:42.337515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.337510    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"
May 26 12:04:42.345173 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.345158    2562 scope.go:117] "RemoveContainer" containerID="2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"
May 26 12:04:42.352477 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.352460    2562 scope.go:117] "RemoveContainer" containerID="8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"
May 26 12:04:42.359346 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.359328    2562 scope.go:117] "RemoveContainer" containerID="07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"
May 26 12:04:42.359600 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:04:42.359578    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623\": container with ID starting with 07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623 not found: ID does not exist" containerID="07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"
May 26 12:04:42.359655 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.359609    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623"} err="failed to get container status \"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623\": rpc error: code = NotFound desc = could not find container \"07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623\": container with ID starting with 07bf3bba069507747a7b30c107023f519e9fcd7ad673505a9f9cfb3819e96623 not found: ID does not exist"
May 26 12:04:42.359655 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.359627    2562 scope.go:117] "RemoveContainer" containerID="2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"
May 26 12:04:42.359853 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:04:42.359836    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a\": container with ID starting with 2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a not found: ID does not exist" containerID="2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"
May 26 12:04:42.359904 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.359856    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a"} err="failed to get container status \"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a\": rpc error: code = NotFound desc = could not find container \"2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a\": container with ID starting with 2ec69c9dc00d0b1ce2f74052eb503456ae076ba30b81e9544fe33c296833a10a not found: ID does not exist"
May 26 12:04:42.359904 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.359869    2562 scope.go:117] "RemoveContainer" containerID="8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"
May 26 12:04:42.360024 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:04:42.360009    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517\": container with ID starting with 8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517 not found: ID does not exist" containerID="8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"
May 26 12:04:42.360064 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.360027    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517"} err="failed to get container status \"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517\": rpc error: code = NotFound desc = could not find container \"8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517\": container with ID starting with 8d968d33a526d1587aded6142e79cbae3d8192903d4dcfd7869db568c083b517 not found: ID does not exist"
May 26 12:04:42.361261 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361244    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g746n\" (UniqueName: \"kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361317 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361272    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361355 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361323    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361470 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361453    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361539 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361520    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361591 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361572    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache\") pod \"722df366-af36-4769-9fb5-8fd5cfd815f1\" (UID: \"722df366-af36-4769-9fb5-8fd5cfd815f1\") "
May 26 12:04:42.361848 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361821    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:04:42.362048 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361878    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:04:42.362048 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.361890    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:04:42.362193 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.362180    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:04:42.363382 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.363357    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:04:42.363497 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.363405    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n" (OuterVolumeSpecName: "kube-api-access-g746n") pod "722df366-af36-4769-9fb5-8fd5cfd815f1" (UID: "722df366-af36-4769-9fb5-8fd5cfd815f1"). InnerVolumeSpecName "kube-api-access-g746n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:04:42.462706 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462671    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g746n\" (UniqueName: \"kubernetes.io/projected/722df366-af36-4769-9fb5-8fd5cfd815f1-kube-api-access-g746n\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.462706 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462701    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.462706 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462714    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722df366-af36-4769-9fb5-8fd5cfd815f1-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.462948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462723    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.462948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462732    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.462948 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.462741    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/722df366-af36-4769-9fb5-8fd5cfd815f1-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:04:42.557393 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.557355    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.35:9003\" within 1s: context deadline exceeded"
May 26 12:04:42.662379 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.662344    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:04:42.667635 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.667609    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-wfxzh"]
May 26 12:04:42.678916 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:42.678871    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" path="/var/lib/kubelet/pods/722df366-af36-4769-9fb5-8fd5cfd815f1/volumes"
May 26 12:04:55.038227 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038188    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:04:55.038806 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038784    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="tokenizer"
May 26 12:04:55.038849 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038812    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="tokenizer"
May 26 12:04:55.038849 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038826    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="main"
May 26 12:04:55.038849 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038836    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="main"
May 26 12:04:55.038954 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038848    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="storage-initializer"
May 26 12:04:55.038954 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038857    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="storage-initializer"
May 26 12:04:55.039020 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038964    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="main"
May 26 12:04:55.039020 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.038977    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="722df366-af36-4769-9fb5-8fd5cfd815f1" containerName="tokenizer"
May 26 12:04:55.043881 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.043857    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.046522 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.046496    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 12:04:55.046652 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.046497    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-d62rw\""
May 26 12:04:55.052257 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.052232    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:04:55.177636 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82tw\" (UniqueName: \"kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.177805 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.177805 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.177805 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.177805 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.177805 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.177802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.278854 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.278817    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.278854 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.278855    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279197 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.278902    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279197 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.278937    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f82tw\" (UniqueName: \"kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279197 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.278992    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279197 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.279018    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279532 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.279286    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279532 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.279325    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279532 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.279445    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.279532 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.279450    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.281547 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.281522    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.291297 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.291243    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82tw\" (UniqueName: \"kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw\") pod \"stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.354198 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.354162    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:55.483603 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:55.483568    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:04:55.487853 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:04:55.487824    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5811498_2934_442f_b24b_2acfc7a8f090.slice/crio-e38cd090ba58d13c56be717d22e845ef6a02c946d12098490a7693fd54109a19 WatchSource:0}: Error finding container e38cd090ba58d13c56be717d22e845ef6a02c946d12098490a7693fd54109a19: Status 404 returned error can't find the container with id e38cd090ba58d13c56be717d22e845ef6a02c946d12098490a7693fd54109a19
May 26 12:04:56.387157 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:56.387120    2562 generic.go:358] "Generic (PLEG): container finished" podID="d5811498-2934-442f-b24b-2acfc7a8f090" containerID="c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c" exitCode=0
May 26 12:04:56.387643 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:56.387183    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerDied","Data":"c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c"}
May 26 12:04:56.387643 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:56.387226    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerStarted","Data":"e38cd090ba58d13c56be717d22e845ef6a02c946d12098490a7693fd54109a19"}
May 26 12:04:57.393780 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:57.393746    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerStarted","Data":"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"}
May 26 12:04:57.393780 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:57.393783    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerStarted","Data":"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"}
May 26 12:04:57.394177 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:57.393890    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:04:57.419198 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:04:57.419150    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" podStartSLOduration=2.419135557 podStartE2EDuration="2.419135557s" podCreationTimestamp="2026-05-26 12:04:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:04:57.416842833 +0000 UTC m=+2393.377390331" watchObservedRunningTime="2026-05-26 12:04:57.419135557 +0000 UTC m=+2393.379683041"
May 26 12:05:04.747252 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:04.747217    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:05:04.751480 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:04.751461    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:05:05.355394 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:05.355352    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:05:05.355394 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:05.355399    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:05:05.357926 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:05.357896    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:05:05.421323 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:05.421209    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:05:26.425625 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:05:26.425552    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:06:36.057128 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:36.057089    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:06:36.059565 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:36.057549    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="main" containerID="cri-o://e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185" gracePeriod=30
May 26 12:06:36.059565 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:36.057566    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="tokenizer" containerID="cri-o://ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9" gracePeriod=30
May 26 12:06:36.424880 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:06:36.424848    2562 logging.go:55] [core] [Channel #1093 SubChannel #1094]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.36:9003", ServerName: "10.134.0.36:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.36:9003: connect: connection refused"
May 26 12:06:36.725254 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:36.725153    2562 generic.go:358] "Generic (PLEG): container finished" podID="d5811498-2934-442f-b24b-2acfc7a8f090" containerID="e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185" exitCode=0
May 26 12:06:36.725254 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:36.725194    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerDied","Data":"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"}
May 26 12:06:37.304535 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.304513    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:06:37.424642 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.424593    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.36:9003\" within 1s: context deadline exceeded"
May 26 12:06:37.470818 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.470778    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.470977 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.470854    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.470977 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.470894    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.470977 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.470936    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.471135 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.470985    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f82tw\" (UniqueName: \"kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.471135 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471010    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache\") pod \"d5811498-2934-442f-b24b-2acfc7a8f090\" (UID: \"d5811498-2934-442f-b24b-2acfc7a8f090\") "
May 26 12:06:37.471242 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471193    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:06:37.471242 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471213    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:06:37.471359 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471288    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.471359 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471305    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.471359 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471347    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:06:37.471793 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.471769    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:06:37.473164 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.473139    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw" (OuterVolumeSpecName: "kube-api-access-f82tw") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "kube-api-access-f82tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:06:37.473164 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.473144    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5811498-2934-442f-b24b-2acfc7a8f090" (UID: "d5811498-2934-442f-b24b-2acfc7a8f090"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:06:37.572712 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.572675    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82tw\" (UniqueName: \"kubernetes.io/projected/d5811498-2934-442f-b24b-2acfc7a8f090-kube-api-access-f82tw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.572712 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.572704    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.572712 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.572714    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5811498-2934-442f-b24b-2acfc7a8f090-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.572712 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.572723    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5811498-2934-442f-b24b-2acfc7a8f090-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:06:37.730552 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.730459    2562 generic.go:358] "Generic (PLEG): container finished" podID="d5811498-2934-442f-b24b-2acfc7a8f090" containerID="ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9" exitCode=0
May 26 12:06:37.730552 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.730534    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerDied","Data":"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"}
May 26 12:06:37.730764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.730576    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4" event={"ID":"d5811498-2934-442f-b24b-2acfc7a8f090","Type":"ContainerDied","Data":"e38cd090ba58d13c56be717d22e845ef6a02c946d12098490a7693fd54109a19"}
May 26 12:06:37.730764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.730591    2562 scope.go:117] "RemoveContainer" containerID="ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"
May 26 12:06:37.730764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.730550    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"
May 26 12:06:37.739861 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.739844    2562 scope.go:117] "RemoveContainer" containerID="e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"
May 26 12:06:37.746855 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.746836    2562 scope.go:117] "RemoveContainer" containerID="c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c"
May 26 12:06:37.753980 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.753961    2562 scope.go:117] "RemoveContainer" containerID="ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"
May 26 12:06:37.754235 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:06:37.754215    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9\": container with ID starting with ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9 not found: ID does not exist" containerID="ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"
May 26 12:06:37.754328 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754248    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9"} err="failed to get container status \"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9\": rpc error: code = NotFound desc = could not find container \"ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9\": container with ID starting with ff4016a61ed7ba22fa432dcfb44abd72ab7da48a2d1a86f982feb50072b78dd9 not found: ID does not exist"
May 26 12:06:37.754328 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754272    2562 scope.go:117] "RemoveContainer" containerID="e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"
May 26 12:06:37.754486 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754459    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:06:37.754653 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:06:37.754632    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185\": container with ID starting with e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185 not found: ID does not exist" containerID="e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"
May 26 12:06:37.754705 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754665    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185"} err="failed to get container status \"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185\": rpc error: code = NotFound desc = could not find container \"e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185\": container with ID starting with e528f89a3b2c5a1a19986906a57aa752f7673b304a6d5a33452cbcbf72ac0185 not found: ID does not exist"
May 26 12:06:37.754705 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754683    2562 scope.go:117] "RemoveContainer" containerID="c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c"
May 26 12:06:37.754920 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:06:37.754904    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c\": container with ID starting with c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c not found: ID does not exist" containerID="c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c"
May 26 12:06:37.754969 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.754923    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c"} err="failed to get container status \"c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c\": rpc error: code = NotFound desc = could not find container \"c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c\": container with ID starting with c681fa0970d99346ed89344c3ef41eb0a21e7046fe58b86b918c9bfc8a780c7c not found: ID does not exist"
May 26 12:06:37.759728 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:37.759704    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954d4b9-lmpm4"]
May 26 12:06:38.678525 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:06:38.678494    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" path="/var/lib/kubelet/pods/d5811498-2934-442f-b24b-2acfc7a8f090/volumes"
May 26 12:10:04.777312 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:04.777286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:10:04.777869 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:04.777702    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:10:16.240438 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:16.240381    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 12:10:16.240980 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:16.240722    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="main" containerID="cri-o://a0489d5b50811a2f22d4c9bc18bf18953f33e235b28f94910ff019b09dec924f" gracePeriod=30
May 26 12:10:16.240980 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:16.240756    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="tokenizer" containerID="cri-o://08a84d1e435d546bd42bfe2394aaac38b6e40fd232c99f3dabb3a52a1733c8a7" gracePeriod=30
May 26 12:10:16.438490 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:16.438458    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerID="a0489d5b50811a2f22d4c9bc18bf18953f33e235b28f94910ff019b09dec924f" exitCode=0
May 26 12:10:16.438636 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:16.438530    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerDied","Data":"a0489d5b50811a2f22d4c9bc18bf18953f33e235b28f94910ff019b09dec924f"}
May 26 12:10:17.443980 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.443944    2562 generic.go:358] "Generic (PLEG): container finished" podID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerID="08a84d1e435d546bd42bfe2394aaac38b6e40fd232c99f3dabb3a52a1733c8a7" exitCode=0
May 26 12:10:17.444371 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.444025    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerDied","Data":"08a84d1e435d546bd42bfe2394aaac38b6e40fd232c99f3dabb3a52a1733c8a7"}
May 26 12:10:17.495735 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.495710    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 12:10:17.641269 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641240    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641479 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641283    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641479 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641340    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641479 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641385    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqh5l\" (UniqueName: \"kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641479 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641406    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641479 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641452    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds\") pod \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\" (UID: \"ca590ad5-2ac8-4f88-a5f8-9956198b0601\") "
May 26 12:10:17.641773 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641649    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:17.641773 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641681    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:17.641773 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.641728    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:17.642150 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.642125    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:17.643532 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.643513    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l" (OuterVolumeSpecName: "kube-api-access-gqh5l") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "kube-api-access-gqh5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:10:17.643604 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.643539    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca590ad5-2ac8-4f88-a5f8-9956198b0601" (UID: "ca590ad5-2ac8-4f88-a5f8-9956198b0601"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:10:17.742010 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.741974    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqh5l\" (UniqueName: \"kubernetes.io/projected/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kube-api-access-gqh5l\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:17.742010 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.742004    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:17.742010 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.742014    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:17.742244 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.742024    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:17.742244 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.742033    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:17.742244 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:17.742042    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca590ad5-2ac8-4f88-a5f8-9956198b0601-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:10:18.448692 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.448657    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr" event={"ID":"ca590ad5-2ac8-4f88-a5f8-9956198b0601","Type":"ContainerDied","Data":"344fe568177de4dfb0d92d4a5ce2833b1bbde1cb59e2a314126cddc6c0db76eb"}
May 26 12:10:18.449150 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.448703    2562 scope.go:117] "RemoveContainer" containerID="08a84d1e435d546bd42bfe2394aaac38b6e40fd232c99f3dabb3a52a1733c8a7"
May 26 12:10:18.449150 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.448673    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"
May 26 12:10:18.457015 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.456997    2562 scope.go:117] "RemoveContainer" containerID="a0489d5b50811a2f22d4c9bc18bf18953f33e235b28f94910ff019b09dec924f"
May 26 12:10:18.463919 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.463901    2562 scope.go:117] "RemoveContainer" containerID="9ddd7e42b8ee621f61df746f0d1ddd12b18da34d4046a603eecb818541ea8774"
May 26 12:10:18.470860 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.470799    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 12:10:18.477816 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.477790    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-86bd76mrzr"]
May 26 12:10:18.678970 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:18.678936    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" path="/var/lib/kubelet/pods/ca590ad5-2ac8-4f88-a5f8-9956198b0601/volumes"
May 26 12:10:39.801959 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.801920    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:10:39.802444 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802408    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="tokenizer"
May 26 12:10:39.802500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802448    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="tokenizer"
May 26 12:10:39.802500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802462    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="main"
May 26 12:10:39.802500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802471    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="main"
May 26 12:10:39.802500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802488    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="storage-initializer"
May 26 12:10:39.802500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802496    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="storage-initializer"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802507    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="storage-initializer"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802517    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="storage-initializer"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802526    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="main"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802536    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="main"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802546    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="tokenizer"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802554    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="tokenizer"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802648    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="main"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802664    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="main"
May 26 12:10:39.802676 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802676    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5811498-2934-442f-b24b-2acfc7a8f090" containerName="tokenizer"
May 26 12:10:39.803224 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.802687    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca590ad5-2ac8-4f88-a5f8-9956198b0601" containerName="tokenizer"
May 26 12:10:39.806248 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.806232    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.810918 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.810892    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 12:10:39.811114 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.811093    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 26 12:10:39.812002 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.811943    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-5xm7j\""
May 26 12:10:39.812002 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.811988    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 12:10:39.812172 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.812016    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qjhlk\""
May 26 12:10:39.823638 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.823614    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:10:39.933217 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.933378 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.933378 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.933378 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgz7\" (UniqueName: \"kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.933537 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:39.933537 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:39.933408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034365 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034329    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgz7\" (UniqueName: \"kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034365 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034367    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034622 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034397    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034622 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034453    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034622 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034479    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034622 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034504    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034819 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034797    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034859 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034815    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034900 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034884    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.034941 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.034904    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.036843 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.036825    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.042806 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.042782    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgz7\" (UniqueName: \"kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.115732 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.115643    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:40.244988 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.244957    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:10:40.247970 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:10:40.247946    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37881f34_8f18_402c_b279_49f5420e3aa7.slice/crio-2a09403a4c88909c9ce012f13cfa2c35ed1a5f29c39a536b7ea281457e3fdeca WatchSource:0}: Error finding container 2a09403a4c88909c9ce012f13cfa2c35ed1a5f29c39a536b7ea281457e3fdeca: Status 404 returned error can't find the container with id 2a09403a4c88909c9ce012f13cfa2c35ed1a5f29c39a536b7ea281457e3fdeca
May 26 12:10:40.249794 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.249779    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:10:40.521568 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.521538    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerStarted","Data":"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"}
May 26 12:10:40.521568 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:40.521572    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerStarted","Data":"2a09403a4c88909c9ce012f13cfa2c35ed1a5f29c39a536b7ea281457e3fdeca"}
May 26 12:10:41.525649 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:41.525613    2562 generic.go:358] "Generic (PLEG): container finished" podID="37881f34-8f18-402c-b279-49f5420e3aa7" containerID="6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830" exitCode=0
May 26 12:10:41.526028 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:41.525703    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerDied","Data":"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"}
May 26 12:10:42.531132 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:42.531094    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerStarted","Data":"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"}
May 26 12:10:42.531132 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:42.531134    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerStarted","Data":"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"}
May 26 12:10:42.531715 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:42.531216    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:42.553315 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:42.553262    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" podStartSLOduration=3.5532464580000003 podStartE2EDuration="3.553246458s" podCreationTimestamp="2026-05-26 12:10:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:10:42.553076217 +0000 UTC m=+2738.513623726" watchObservedRunningTime="2026-05-26 12:10:42.553246458 +0000 UTC m=+2738.513793943"
May 26 12:10:50.116571 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:50.116532    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:50.116571 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:50.116580    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:50.119500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:50.119475    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:10:50.560502 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:10:50.560474    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:11:11.564189 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:11.564158    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:11:18.418617 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.418513    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:11:18.423109 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.423085    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.425738 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.425710    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-z5h7v\""
May 26 12:11:18.425914 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.425887    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 26 12:11:18.433671 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.433621    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:11:18.557515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.557515 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.557764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.557764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwtw\" (UniqueName: \"kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.557764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.557764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.557732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.658771 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658738    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.658771 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658772    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659019 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658794    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659019 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658813    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwtw\" (UniqueName: \"kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659019 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658850    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659019 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.658958    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659241 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.659177    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659303 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.659235    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659303 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.659242    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.659374 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.659311    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.661260 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.661239    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.669204 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.669142    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwtw\" (UniqueName: \"kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.735804 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.735775    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:18.863275 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:18.863234    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:11:18.866460 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:11:18.866433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7497ac_a38c_452b_95d6_23ae213690fd.slice/crio-6eba111de145bc5a0e388048a05d73dbc8c7bf517c69adc3d8b8b753770fabcc WatchSource:0}: Error finding container 6eba111de145bc5a0e388048a05d73dbc8c7bf517c69adc3d8b8b753770fabcc: Status 404 returned error can't find the container with id 6eba111de145bc5a0e388048a05d73dbc8c7bf517c69adc3d8b8b753770fabcc
May 26 12:11:19.659568 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:19.659535    2562 generic.go:358] "Generic (PLEG): container finished" podID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerID="94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb" exitCode=0
May 26 12:11:19.659927 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:19.659608    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerDied","Data":"94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb"}
May 26 12:11:19.659927 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:19.659630    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerStarted","Data":"6eba111de145bc5a0e388048a05d73dbc8c7bf517c69adc3d8b8b753770fabcc"}
May 26 12:11:20.665296 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:20.665265    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerStarted","Data":"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"}
May 26 12:11:20.665731 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:20.665304    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerStarted","Data":"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"}
May 26 12:11:20.665731 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:20.665438    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:20.688541 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:20.688488    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" podStartSLOduration=2.688470466 podStartE2EDuration="2.688470466s" podCreationTimestamp="2026-05-26 12:11:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:11:20.686949704 +0000 UTC m=+2776.647497189" watchObservedRunningTime="2026-05-26 12:11:20.688470466 +0000 UTC m=+2776.649017952"
May 26 12:11:28.736819 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:28.736787    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:28.736819 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:28.736828    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:28.739631 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:28.739604    2562 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:29.696504 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:29.696476    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:11:50.699205 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:11:50.699176    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:13:59.160375 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:13:59.160339    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:13:59.160989 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:13:59.160737    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="main" containerID="cri-o://dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e" gracePeriod=30
May 26 12:13:59.160989 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:13:59.160784    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="tokenizer" containerID="cri-o://a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7" gracePeriod=30
May 26 12:13:59.696239 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:13:59.696192    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.38:8082/healthz\": dial tcp 10.134.0.38:8082: connect: connection refused"
May 26 12:14:00.199548 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.199521    2562 generic.go:358] "Generic (PLEG): container finished" podID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerID="dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e" exitCode=0
May 26 12:14:00.199893 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.199588    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerDied","Data":"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"}
May 26 12:14:00.412520 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.412495    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:14:00.453841 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453765    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.453841 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453801    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.453841 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453822    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.454116 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453845    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfwtw\" (UniqueName: \"kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.454116 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453885    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.454116 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.453911    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location\") pod \"7a7497ac-a38c-452b-95d6-23ae213690fd\" (UID: \"7a7497ac-a38c-452b-95d6-23ae213690fd\") "
May 26 12:14:00.454116 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.454096    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.454116 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.454107    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.454396 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.454350    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.454671 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.454646    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.456048 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.456021    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw" (OuterVolumeSpecName: "kube-api-access-lfwtw") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "kube-api-access-lfwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:00.456160 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.456135    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a7497ac-a38c-452b-95d6-23ae213690fd" (UID: "7a7497ac-a38c-452b-95d6-23ae213690fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:00.554786 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554753    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:00.554786 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554783    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:00.554786 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554793    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:00.555017 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554802    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7497ac-a38c-452b-95d6-23ae213690fd-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:00.555017 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554811    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7497ac-a38c-452b-95d6-23ae213690fd-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:00.555017 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:00.554820    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwtw\" (UniqueName: \"kubernetes.io/projected/7a7497ac-a38c-452b-95d6-23ae213690fd-kube-api-access-lfwtw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:01.205008 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.204975    2562 generic.go:358] "Generic (PLEG): container finished" podID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerID="a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7" exitCode=0
May 26 12:14:01.205499 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.205042    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerDied","Data":"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"}
May 26 12:14:01.205499 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.205048    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"
May 26 12:14:01.205499 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.205075    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9" event={"ID":"7a7497ac-a38c-452b-95d6-23ae213690fd","Type":"ContainerDied","Data":"6eba111de145bc5a0e388048a05d73dbc8c7bf517c69adc3d8b8b753770fabcc"}
May 26 12:14:01.205499 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.205096    2562 scope.go:117] "RemoveContainer" containerID="a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"
May 26 12:14:01.213046 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.213030    2562 scope.go:117] "RemoveContainer" containerID="dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"
May 26 12:14:01.220088 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.220074    2562 scope.go:117] "RemoveContainer" containerID="94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb"
May 26 12:14:01.230098 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.230077    2562 scope.go:117] "RemoveContainer" containerID="a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"
May 26 12:14:01.230753 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:01.230717    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7\": container with ID starting with a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7 not found: ID does not exist" containerID="a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"
May 26 12:14:01.230843 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.230758    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7"} err="failed to get container status \"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7\": rpc error: code = NotFound desc = could not find container \"a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7\": container with ID starting with a2bc976a349a0e6d2f5d3cd32d3e78c77647d54a3d51e71989b939a6270cd3b7 not found: ID does not exist"
May 26 12:14:01.230843 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.230783    2562 scope.go:117] "RemoveContainer" containerID="dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"
May 26 12:14:01.231067 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:01.231044    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e\": container with ID starting with dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e not found: ID does not exist" containerID="dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"
May 26 12:14:01.231123 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.231074    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e"} err="failed to get container status \"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e\": rpc error: code = NotFound desc = could not find container \"dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e\": container with ID starting with dda8bed11a5483f8ddfa02127cb22a509bf5906983fedb7add3be6231d92a71e not found: ID does not exist"
May 26 12:14:01.231123 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.231096    2562 scope.go:117] "RemoveContainer" containerID="94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb"
May 26 12:14:01.231694 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:01.231666    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb\": container with ID starting with 94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb not found: ID does not exist" containerID="94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb"
May 26 12:14:01.231785 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.231702    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb"} err="failed to get container status \"94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb\": rpc error: code = NotFound desc = could not find container \"94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb\": container with ID starting with 94077d6844a1bb420959f8e99e63d533882069dc591c84ffcf725b9d780f8bdb not found: ID does not exist"
May 26 12:14:01.232667 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.232646    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:14:01.235457 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:01.235435    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheq2tp9"]
May 26 12:14:02.679013 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:02.678974    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" path="/var/lib/kubelet/pods/7a7497ac-a38c-452b-95d6-23ae213690fd/volumes"
May 26 12:14:04.903195 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:04.903155    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:14:04.903699 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:04.903495    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="main" containerID="cri-o://fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9" gracePeriod=30
May 26 12:14:04.903699 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:04.903552    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="tokenizer" containerID="cri-o://16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4" gracePeriod=30
May 26 12:14:05.225715 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:05.225610    2562 generic.go:358] "Generic (PLEG): container finished" podID="37881f34-8f18-402c-b279-49f5420e3aa7" containerID="fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9" exitCode=0
May 26 12:14:05.225715 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:05.225683    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerDied","Data":"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"}
May 26 12:14:06.154146 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.154122    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:14:06.200393 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200312    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200393 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200373    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200598 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200462    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjgz7\" (UniqueName: \"kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200598 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200540    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200598 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200567    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200756 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200619    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache\") pod \"37881f34-8f18-402c-b279-49f5420e3aa7\" (UID: \"37881f34-8f18-402c-b279-49f5420e3aa7\") "
May 26 12:14:06.200756 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200681    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:06.200891 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200867    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:06.200958 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200877    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-tmp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.201010 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.200989    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:06.201607 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.201580    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:06.202618 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.202599    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:06.202764 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.202738    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7" (OuterVolumeSpecName: "kube-api-access-fjgz7") pod "37881f34-8f18-402c-b279-49f5420e3aa7" (UID: "37881f34-8f18-402c-b279-49f5420e3aa7"). InnerVolumeSpecName "kube-api-access-fjgz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:06.229982 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.229950    2562 generic.go:358] "Generic (PLEG): container finished" podID="37881f34-8f18-402c-b279-49f5420e3aa7" containerID="16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4" exitCode=0
May 26 12:14:06.230122 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.230019    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerDied","Data":"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"}
May 26 12:14:06.230122 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.230041    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx" event={"ID":"37881f34-8f18-402c-b279-49f5420e3aa7","Type":"ContainerDied","Data":"2a09403a4c88909c9ce012f13cfa2c35ed1a5f29c39a536b7ea281457e3fdeca"}
May 26 12:14:06.230122 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.230046    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"
May 26 12:14:06.230122 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.230058    2562 scope.go:117] "RemoveContainer" containerID="16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"
May 26 12:14:06.238091 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.238055    2562 scope.go:117] "RemoveContainer" containerID="fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"
May 26 12:14:06.245050 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.245035    2562 scope.go:117] "RemoveContainer" containerID="6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"
May 26 12:14:06.252022 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.252008    2562 scope.go:117] "RemoveContainer" containerID="16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"
May 26 12:14:06.252292 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:06.252263    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4\": container with ID starting with 16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4 not found: ID does not exist" containerID="16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"
May 26 12:14:06.252407 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.252299    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4"} err="failed to get container status \"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4\": rpc error: code = NotFound desc = could not find container \"16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4\": container with ID starting with 16382e11eb27923d3ad36537647c4e85e21563fe9712e3460c9b6c9e7f0eb6c4 not found: ID does not exist"
May 26 12:14:06.252407 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.252319    2562 scope.go:117] "RemoveContainer" containerID="fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"
May 26 12:14:06.252738 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:06.252678    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9\": container with ID starting with fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9 not found: ID does not exist" containerID="fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"
May 26 12:14:06.252738 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.252723    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9"} err="failed to get container status \"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9\": rpc error: code = NotFound desc = could not find container \"fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9\": container with ID starting with fc8fdac1b670f5bc15e797f410da78f475da671b2873ede06818d94942ff2db9 not found: ID does not exist"
May 26 12:14:06.252971 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.252749    2562 scope.go:117] "RemoveContainer" containerID="6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"
May 26 12:14:06.253085 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:06.253062    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830\": container with ID starting with 6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830 not found: ID does not exist" containerID="6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"
May 26 12:14:06.253148 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.253093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830"} err="failed to get container status \"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830\": rpc error: code = NotFound desc = could not find container \"6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830\": container with ID starting with 6249baa890b80fadfc5ee16616c0120107cf36f5023318cc705e7eb845ae4830 not found: ID does not exist"
May 26 12:14:06.254548 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.254527    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:14:06.259047 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.259026    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-75b4cd78fkfhtx"]
May 26 12:14:06.301285 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.301258    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-uds\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.301285 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.301286    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-kserve-provision-location\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.301469 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.301301    2562 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/37881f34-8f18-402c-b279-49f5420e3aa7-tokenizer-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.301469 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.301312    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37881f34-8f18-402c-b279-49f5420e3aa7-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.301469 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.301321    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgz7\" (UniqueName: \"kubernetes.io/projected/37881f34-8f18-402c-b279-49f5420e3aa7-kube-api-access-fjgz7\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:06.322553 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322526    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgl4n/must-gather-qh2v5"]
May 26 12:14:06.322850 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322836    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="storage-initializer"
May 26 12:14:06.322891 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322852    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="storage-initializer"
May 26 12:14:06.322891 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322862    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="main"
May 26 12:14:06.322891 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322868    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="main"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322892    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="main"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322898    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="main"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322904    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="tokenizer"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322910    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="tokenizer"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322919    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="tokenizer"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322925    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="tokenizer"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322932    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="storage-initializer"
May 26 12:14:06.322983 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.322937    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="storage-initializer"
May 26 12:14:06.323209 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.323007    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="main"
May 26 12:14:06.323209 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.323015    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="main"
May 26 12:14:06.323209 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.323023    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="7a7497ac-a38c-452b-95d6-23ae213690fd" containerName="tokenizer"
May 26 12:14:06.323209 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.323031    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" containerName="tokenizer"
May 26 12:14:06.327428 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.327400    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.333893 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.333770    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgl4n\"/\"openshift-service-ca.crt\""
May 26 12:14:06.333893 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.333875    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgl4n\"/\"kube-root-ca.crt\""
May 26 12:14:06.334101 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.334056    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgl4n\"/\"default-dockercfg-vn6db\""
May 26 12:14:06.340997 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.340977    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgl4n/must-gather-qh2v5"]
May 26 12:14:06.402366 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.402336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.402550 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.402433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vjw\" (UniqueName: \"kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.503873 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.503776    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vjw\" (UniqueName: \"kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.503873 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.503863    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.504169 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.504155    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.512022 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.511993    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vjw\" (UniqueName: \"kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw\") pod \"must-gather-qh2v5\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") " pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.636335 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.636294    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:06.678527 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.678497    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37881f34-8f18-402c-b279-49f5420e3aa7" path="/var/lib/kubelet/pods/37881f34-8f18-402c-b279-49f5420e3aa7/volumes"
May 26 12:14:06.760482 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:06.760375    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgl4n/must-gather-qh2v5"]
May 26 12:14:06.763275 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:14:06.763252    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d0e3bf_78cc_4d9a_b57b_2ff79c7c6a3d.slice/crio-8cd31317928398f5da8c9497904ad7cdfba2bbcc6cd9d0578a0f89c549b49d67 WatchSource:0}: Error finding container 8cd31317928398f5da8c9497904ad7cdfba2bbcc6cd9d0578a0f89c549b49d67: Status 404 returned error can't find the container with id 8cd31317928398f5da8c9497904ad7cdfba2bbcc6cd9d0578a0f89c549b49d67
May 26 12:14:07.234185 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:07.234150    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" event={"ID":"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d","Type":"ContainerStarted","Data":"8cd31317928398f5da8c9497904ad7cdfba2bbcc6cd9d0578a0f89c549b49d67"}
May 26 12:14:13.266751 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:13.266667    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" event={"ID":"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d","Type":"ContainerStarted","Data":"c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530"}
May 26 12:14:13.266751 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:13.266704    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" event={"ID":"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d","Type":"ContainerStarted","Data":"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"}
May 26 12:14:13.286740 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:13.286694    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" podStartSLOduration=1.063697614 podStartE2EDuration="7.286678918s" podCreationTimestamp="2026-05-26 12:14:06 +0000 UTC" firstStartedPulling="2026-05-26 12:14:06.764914099 +0000 UTC m=+2942.725461561" lastFinishedPulling="2026-05-26 12:14:12.9878954 +0000 UTC m=+2948.948442865" observedRunningTime="2026-05-26 12:14:13.285457236 +0000 UTC m=+2949.246004724" watchObservedRunningTime="2026-05-26 12:14:13.286678918 +0000 UTC m=+2949.247226403"
May 26 12:14:39.744500 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:39.744462    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4bgz6_a9cf80bb-d98c-494b-9445-ce491eaa5e32/authorino/0.log"
May 26 12:14:39.800637 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:39.800609    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-djxwj_23ec7f8a-039e-49d5-b67a-5156bd10c9b2/manager/0.log"
May 26 12:14:39.919554 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:39.919526    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-892hk_9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a/manager/0.log"
May 26 12:14:41.376765 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:41.376733    2562 generic.go:358] "Generic (PLEG): container finished" podID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerID="8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2" exitCode=0
May 26 12:14:41.377229 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:41.376779    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" event={"ID":"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d","Type":"ContainerDied","Data":"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"}
May 26 12:14:41.377229 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:41.377095    2562 scope.go:117] "RemoveContainer" containerID="8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"
May 26 12:14:41.563753 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:41.563718    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hgl4n_must-gather-qh2v5_70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d/gather/0.log"
May 26 12:14:45.454142 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:45.452475    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z5xsh_217e3e9a-400f-4c89-aed7-d005b5a51e76/global-pull-secret-syncer/0.log"
May 26 12:14:45.491322 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:45.491262    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bxkxv_c466082b-97a7-404a-91c0-8c9ac1e9c791/konnectivity-agent/0.log"
May 26 12:14:45.602376 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:45.602322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-125.ec2.internal_f88796f630f0b1df3d5abb08c2fec007/haproxy/0.log"
May 26 12:14:47.126367 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.126333    2562 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hgl4n/must-gather-qh2v5"]
May 26 12:14:47.126796 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.126571    2562 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="copy" containerID="cri-o://c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530" gracePeriod=2
May 26 12:14:47.134449 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.134405    2562 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hgl4n/must-gather-qh2v5"]
May 26 12:14:47.355319 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.355298    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hgl4n_must-gather-qh2v5_70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d/copy/0.log"
May 26 12:14:47.355728 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.355712    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:47.357979 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.357955    2562 status_manager.go:895] "Failed to get status for pod" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" err="pods \"must-gather-qh2v5\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hgl4n\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object"
May 26 12:14:47.361317 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.361302    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output\") pod \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") "
May 26 12:14:47.361375 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.361341    2562 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6vjw\" (UniqueName: \"kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw\") pod \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\" (UID: \"70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d\") "
May 26 12:14:47.363434 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.363390    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw" (OuterVolumeSpecName: "kube-api-access-b6vjw") pod "70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" (UID: "70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d"). InnerVolumeSpecName "kube-api-access-b6vjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:47.367646 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.367624    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" (UID: "70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:47.396134 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.396073    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hgl4n_must-gather-qh2v5_70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d/copy/0.log"
May 26 12:14:47.396385 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.396362    2562 generic.go:358] "Generic (PLEG): container finished" podID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerID="c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530" exitCode=143
May 26 12:14:47.396471 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.396433    2562 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgl4n/must-gather-qh2v5"
May 26 12:14:47.396513 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.396471    2562 scope.go:117] "RemoveContainer" containerID="c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530"
May 26 12:14:47.399109 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.399083    2562 status_manager.go:895] "Failed to get status for pod" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" err="pods \"must-gather-qh2v5\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hgl4n\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object"
May 26 12:14:47.403888 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.403861    2562 scope.go:117] "RemoveContainer" containerID="8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"
May 26 12:14:47.407305 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.407279    2562 status_manager.go:895] "Failed to get status for pod" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" pod="openshift-must-gather-hgl4n/must-gather-qh2v5" err="pods \"must-gather-qh2v5\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hgl4n\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object"
May 26 12:14:47.417117 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.417101    2562 scope.go:117] "RemoveContainer" containerID="c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530"
May 26 12:14:47.417356 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:47.417336    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530\": container with ID starting with c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530 not found: ID does not exist" containerID="c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530"
May 26 12:14:47.417440 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.417362    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530"} err="failed to get container status \"c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530\": rpc error: code = NotFound desc = could not find container \"c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530\": container with ID starting with c732c5c7289d30ecdb0406b368e9d9eb14fa3d7e6fbd749ea3b99bc1c88be530 not found: ID does not exist"
May 26 12:14:47.417440 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.417382    2562 scope.go:117] "RemoveContainer" containerID="8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"
May 26 12:14:47.417608 ip-10-0-133-125 kubenswrapper[2562]: E0526 12:14:47.417589    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2\": container with ID starting with 8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2 not found: ID does not exist" containerID="8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"
May 26 12:14:47.417649 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.417613    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2"} err="failed to get container status \"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2\": rpc error: code = NotFound desc = could not find container \"8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2\": container with ID starting with 8f9f69ecda25b913a007dfbc27513c9e8f6426d7bbecd689039c3172602493c2 not found: ID does not exist"
May 26 12:14:47.462545 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.462518    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-must-gather-output\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:47.462545 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:47.462542    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6vjw\" (UniqueName: \"kubernetes.io/projected/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d-kube-api-access-b6vjw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
May 26 12:14:48.678865 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:48.678830    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" path="/var/lib/kubelet/pods/70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d/volumes"
May 26 12:14:49.529782 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:49.529755    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-4bgz6_a9cf80bb-d98c-494b-9445-ce491eaa5e32/authorino/0.log"
May 26 12:14:49.594322 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:49.594298    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-djxwj_23ec7f8a-039e-49d5-b67a-5156bd10c9b2/manager/0.log"
May 26 12:14:49.794600 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:49.794518    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-892hk_9e4e27d7-09ed-49d3-8d4e-f29d09b3c48a/manager/0.log"
May 26 12:14:51.286128 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.286089    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fcc9475df-q4q8c_50c18f3f-f750-40ef-b5cf-db3beb602df5/metrics-server/0.log"
May 26 12:14:51.316271 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.316190    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-5ljjc_9e6deeb7-8f24-445d-bfc8-a5a1fbd4c245/monitoring-plugin/0.log"
May 26 12:14:51.640720 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.640694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8k6h_1cb83698-8cf8-40ad-b882-54745cdf375f/node-exporter/0.log"
May 26 12:14:51.671821 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.671799    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8k6h_1cb83698-8cf8-40ad-b882-54745cdf375f/kube-rbac-proxy/0.log"
May 26 12:14:51.702933 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.702905    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8k6h_1cb83698-8cf8-40ad-b882-54745cdf375f/init-textfile/0.log"
May 26 12:14:51.883702 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.883676    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/prometheus/0.log"
May 26 12:14:51.917330 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.917252    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/config-reloader/0.log"
May 26 12:14:51.949157 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.949132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/thanos-sidecar/0.log"
May 26 12:14:51.984687 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:51.984658    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/kube-rbac-proxy-web/0.log"
May 26 12:14:52.015131 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.015103    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/kube-rbac-proxy/0.log"
May 26 12:14:52.054351 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.054326    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/kube-rbac-proxy-thanos/0.log"
May 26 12:14:52.088588 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.088563    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f839d69a-1168-4d62-936d-a8c6bed97d6e/init-config-reloader/0.log"
May 26 12:14:52.392977 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.392947    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/thanos-query/0.log"
May 26 12:14:52.429034 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.428959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/kube-rbac-proxy-web/0.log"
May 26 12:14:52.472387 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.472358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/kube-rbac-proxy/0.log"
May 26 12:14:52.507046 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.507015    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/prom-label-proxy/0.log"
May 26 12:14:52.560471 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.560440    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/kube-rbac-proxy-rules/0.log"
May 26 12:14:52.602727 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:52.602698    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56588f9995-tb5j5_67a60299-e747-474a-b063-70c60118e2b8/kube-rbac-proxy-metrics/0.log"
May 26 12:14:53.835712 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.835680    2562 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"]
May 26 12:14:53.836169 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="gather"
May 26 12:14:53.836169 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836094    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="gather"
May 26 12:14:53.836169 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836107    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="copy"
May 26 12:14:53.836169 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836115    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="copy"
May 26 12:14:53.836344 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836190    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="gather"
May 26 12:14:53.836344 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.836205    2562 memory_manager.go:356] "RemoveStaleState removing state" podUID="70d0e3bf-78cc-4d9a-b57b-2ff79c7c6a3d" containerName="copy"
May 26 12:14:53.839999 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.839979    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:53.842590 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.842567    2562 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wtzdl\"/\"default-dockercfg-cpmnp\""
May 26 12:14:53.843482 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.843466    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wtzdl\"/\"openshift-service-ca.crt\""
May 26 12:14:53.843541 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.843508    2562 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wtzdl\"/\"kube-root-ca.crt\""
May 26 12:14:53.863190 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.863166    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"]
May 26 12:14:53.911992 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.911964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxv9\" (UniqueName: \"kubernetes.io/projected/af184a95-e730-4c8e-a176-0c2449a6c9fa-kube-api-access-lnxv9\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:53.912136 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.911996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-podres\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:53.912136 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.912018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-lib-modules\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:53.912136 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.912038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-sys\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:53.912248 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:53.912145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-proc\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013435 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013365    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxv9\" (UniqueName: \"kubernetes.io/projected/af184a95-e730-4c8e-a176-0c2449a6c9fa-kube-api-access-lnxv9\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013435 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013448    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-podres\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013656 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013479    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-lib-modules\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013656 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013500    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-sys\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013656 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013585    2562 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-proc\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013656 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013621    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-lib-modules\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013656 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013621    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-podres\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013833 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013654    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-sys\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.013833 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.013671    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af184a95-e730-4c8e-a176-0c2449a6c9fa-proc\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.023091 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.023070    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxv9\" (UniqueName: \"kubernetes.io/projected/af184a95-e730-4c8e-a176-0c2449a6c9fa-kube-api-access-lnxv9\") pod \"perf-node-gather-daemonset-qq54c\" (UID: \"af184a95-e730-4c8e-a176-0c2449a6c9fa\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.149887 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.149855    2562 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.273871 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.273839    2562 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"]
May 26 12:14:54.276065 ip-10-0-133-125 kubenswrapper[2562]: W0526 12:14:54.276037    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf184a95_e730_4c8e_a176_0c2449a6c9fa.slice/crio-2760b4606ba905012444939b6c2a5cc4b7e3917b5c5ae82461a4cbf88efe2015 WatchSource:0}: Error finding container 2760b4606ba905012444939b6c2a5cc4b7e3917b5c5ae82461a4cbf88efe2015: Status 404 returned error can't find the container with id 2760b4606ba905012444939b6c2a5cc4b7e3917b5c5ae82461a4cbf88efe2015
May 26 12:14:54.421147 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.421080    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c" event={"ID":"af184a95-e730-4c8e-a176-0c2449a6c9fa","Type":"ContainerStarted","Data":"905cc0968ed7c72ec145ab56462d4edc2c7006606dcfa08d57aaf9d8a13e44e7"}
May 26 12:14:54.421147 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.421112    2562 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c" event={"ID":"af184a95-e730-4c8e-a176-0c2449a6c9fa","Type":"ContainerStarted","Data":"2760b4606ba905012444939b6c2a5cc4b7e3917b5c5ae82461a4cbf88efe2015"}
May 26 12:14:54.421311 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.421219    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:14:54.440959 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:54.440911    2562 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c" podStartSLOduration=1.440899626 podStartE2EDuration="1.440899626s" podCreationTimestamp="2026-05-26 12:14:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:14:54.438761297 +0000 UTC m=+2990.399308782" watchObservedRunningTime="2026-05-26 12:14:54.440899626 +0000 UTC m=+2990.401447111"
May 26 12:14:55.820650 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:55.820620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ntbs_b140c9d9-dd67-4b37-9745-557d148045d9/dns/0.log"
May 26 12:14:55.845744 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:55.845720    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ntbs_b140c9d9-dd67-4b37-9745-557d148045d9/kube-rbac-proxy/0.log"
May 26 12:14:56.030492 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:56.030463    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lsn6g_c6197213-20ec-4de9-8308-3f48b0cee933/dns-node-resolver/0.log"
May 26 12:14:56.656687 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:56.656651    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8m2x_5cb390ce-0700-4745-9dda-fe590c8284c2/node-ca/0.log"
May 26 12:14:58.303203 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:58.303171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9v7t_817d46b4-0ea1-407a-b72e-7b9292c0a3b8/serve-healthcheck-canary/0.log"
May 26 12:14:58.784768 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:58.784729    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m74wn_0c9d899d-fe8b-4346-bf59-255ebb4f643f/kube-rbac-proxy/0.log"
May 26 12:14:58.807286 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:58.807243    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m74wn_0c9d899d-fe8b-4346-bf59-255ebb4f643f/exporter/0.log"
May 26 12:14:58.829318 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:14:58.829290    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m74wn_0c9d899d-fe8b-4346-bf59-255ebb4f643f/extractor/0.log"
May 26 12:15:00.434582 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:00.434551    2562 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-qq54c"
May 26 12:15:01.582155 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:01.582066    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b8db67594-pmh7r_26678c67-074c-4929-80ed-2f6b04c5575a/manager/0.log"
May 26 12:15:01.640964 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:01.640934    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-zcxr5_95654c47-2165-41e2-9a98-5944bd332d03/openshift-lws-operator/0.log"
May 26 12:15:02.621247 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:02.621206    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-f6bj8_45e57a4d-3e30-4914-b0be-337c1650bf2e/manager/0.log"
May 26 12:15:02.743267 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:02.743242    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-pmqpb_803ead6e-08fb-456a-9779-f36022f916a7/seaweedfs/0.log"
May 26 12:15:04.801433 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:04.801389    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:15:04.804245 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:04.804223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b8trm_fb187e2a-b37a-48ad-8f06-2f74b6bc867e/ovn-acl-logging/0.log"
May 26 12:15:09.043300 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.043263    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/kube-multus-additional-cni-plugins/0.log"
May 26 12:15:09.068256 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.068231    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/egress-router-binary-copy/0.log"
May 26 12:15:09.094280 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.094252    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/cni-plugins/0.log"
May 26 12:15:09.118060 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.118035    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/bond-cni-plugin/0.log"
May 26 12:15:09.141459 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.141411    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/routeoverride-cni/0.log"
May 26 12:15:09.167234 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.167205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/whereabouts-cni-bincopy/0.log"
May 26 12:15:09.189747 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.189724    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z2bzc_b405a8c5-b718-4ec6-bebe-2f1209fe5859/whereabouts-cni/0.log"
May 26 12:15:09.253889 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.253859    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cjv66_50f2f01d-8199-4e84-ab1b-96c4a1a06803/kube-multus/0.log"
May 26 12:15:09.371146 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.371069    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hkf6m_f8bcaf28-492e-42b7-9252-e3cfa4f217f9/network-metrics-daemon/0.log"
May 26 12:15:09.403882 ip-10-0-133-125 kubenswrapper[2562]: I0526 12:15:09.403852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hkf6m_f8bcaf28-492e-42b7-9252-e3cfa4f217f9/kube-rbac-proxy/0.log"