Sep 09 20:53:01.854140 ip-10-0-138-22 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.263949    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268472    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268483    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268487    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268490    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268494    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268496    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268499    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.363269 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268502    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.329894 ip-10-0-138-22 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268504    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268507    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268509    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268514    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268521    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268524    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268528    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268531    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268534    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268537    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268540    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268543    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268546    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268548    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268551    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268554    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268557    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.366499 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268560    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268562    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268565    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268571    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268573    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268576    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268578    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268581    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268583    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268586    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268588    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268591    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268595    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268599    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268602    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268604    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268607    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268609    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268612    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.553854 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268615    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268617    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268620    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268622    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268625    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268627    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268630    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268632    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268635    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268638    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268641    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268644    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268648    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268651    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268654    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268656    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268660    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268664    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268667    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.555238 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268670    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268673    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268676    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268678    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268681    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268684    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268686    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268689    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268692    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268694    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268698    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268701    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268703    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268706    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268708    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268710    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268713    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268715    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268718    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268720    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.556247 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268723    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268725    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268728    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268730    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268733    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268736    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268738    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268741    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268744    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268746    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268749    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268752    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268754    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.268757    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269446    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269454    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269457    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269460    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269463    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.557064 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269466    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269469    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269472    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269475    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269478    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269480    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269483    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269486    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269488    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269491    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269493    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269496    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269499    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269502    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269504    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269507    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269509    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269512    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269515    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269518    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.557693 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269521    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269524    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269526    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269529    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269531    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269534    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269536    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269539    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269542    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269544    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269548    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269552    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269555    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269558    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269561    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269564    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269567    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269569    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269572    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.558438 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269575    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269578    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269582    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269586    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269589    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269591    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269595    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269599    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269602    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269605    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269608    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269611    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269614    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269616    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269619    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269624    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269627    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269629    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269632    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.559077 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269635    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269638    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269640    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269643    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269645    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269648    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269651    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269653    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269657    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269659    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269662    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269664    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269668    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269671    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269673    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269676    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269679    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269681    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269684    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269686    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.559894 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269689    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269692    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269694    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269697    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269700    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269702    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269705    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269707    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269711    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269714    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269717    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269720    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269722    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269725    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269727    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.269730    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270103    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270108    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270111    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270114    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.561484 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270116    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270119    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270123    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270127    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270130    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270133    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270137    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270139    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270142    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270145    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270147    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270150    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270153    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270155    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270158    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270161    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270163    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270166    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270168    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270171    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.562119 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270175    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270178    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270180    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270183    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270186    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270188    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270191    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270193    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270196    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270198    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270201    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270204    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270206    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270209    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270211    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270214    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270217    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270220    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270223    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270227    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.562653 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270230    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270233    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270235    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270238    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270240    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270243    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270245    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270248    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270251    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270254    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270256    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270259    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270263    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270265    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270268    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270270    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270273    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270275    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270278    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270280    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.563378 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270283    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270286    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270289    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270291    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270294    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270296    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270300    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270303    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270306    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270308    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270314    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270317    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270319    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270322    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270324    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270326    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270329    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270331    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270334    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.563945 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270338    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270340    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270343    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270345    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270350    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270353    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270355    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270358    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270361    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270363    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270365    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270368    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270371    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270373    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.270376    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270445    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270478    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270484    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270489    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270493    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270496    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:53:02.564442 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270500    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270504    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270508    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270511    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270514    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270517    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270520    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270523    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270526    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270528    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270531    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270536    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270539    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270542    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270545    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270549    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270553    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270556    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270559    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270562    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270565    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270569    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270571    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270575    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:53:02.564958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270578    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270581    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270584    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270587    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270590    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270593    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270596    2563 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270599    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270603    2563 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270606    2563 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270609    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270612    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270615    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270619    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270622    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270624    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270627    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270630    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270633    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270637    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270640    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270643    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270646    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270650    2563 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270653    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:53:02.565527 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270656    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270659    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270662    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270665    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270668    2563 flags.go:64] FLAG: --help="false"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270671    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270675    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270677    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270680    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270683    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270687    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270690    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270692    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270695    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270698    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270700    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270704    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270706    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270709    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270712    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270715    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270718    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270721    2563 flags.go:64] FLAG: --lock-file=""
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270724    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:53:02.566153 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270727    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270730    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270736    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270739    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270741    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270744    2563 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270748    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270751    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270754    2563 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270757    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270761    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270764    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270768    2563 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270771    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270774    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270777    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270780    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270783    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270786    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270789    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270796    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270799    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270802    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:53:02.566732 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270805    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270808    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270813    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270816    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270819    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270822    2563 flags.go:64] FLAG: --port="10250"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270825    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270828    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02190a8284cc05038"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270831    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270834    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270837    2563 flags.go:64] FLAG: --register-node="true"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270839    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270844    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270847    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270849    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270853    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270857    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270873    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270876    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270879    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270881    2563 flags.go:64] FLAG: --runonce="false"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270884    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270887    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270890    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270893    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:53:02.567289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270896    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270899    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270902    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270905    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270908    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270910    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270913    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270916    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270919    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270922    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270924    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270930    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270932    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270935    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270939    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270942    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270944    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270947    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270950    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270954    2563 flags.go:64] FLAG: --v="2"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270958    2563 flags.go:64] FLAG: --version="false"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270961    2563 flags.go:64] FLAG: --vmodule=""
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270966    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.270970    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:53:02.568161 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271073    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271077    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271080    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271083    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271085    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271088    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271091    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271093    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271096    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271099    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271101    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271104    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271107    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271110    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271112    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271115    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271117    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271120    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271123    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271125    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.568822 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271128    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271130    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271133    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271136    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271138    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271140    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271144    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271149    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271152    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271155    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271160    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271163    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271165    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271168    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271171    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271173    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271176    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271178    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271181    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271184    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.569365 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271186    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271189    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271205    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271209    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271213    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271216    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271218    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271221    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271224    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271227    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271230    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271233    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271235    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271237    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271240    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271243    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271246    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271248    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271251    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271255    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.569921 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271257    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271260    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271265    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271268    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271271    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271274    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271276    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271279    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271282    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271284    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271287    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271289    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271292    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271295    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271297    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271300    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271302    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271305    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271308    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.570400 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271310    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271313    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271315    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271318    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271321    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271323    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271325    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271328    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271332    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271337    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271340    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271343    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271347    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271350    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271353    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271359    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271361    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271364    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271366    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.570853 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.271377    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.272522    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.279217    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.279230    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.279249    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.279259    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279311    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279315    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279319    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279322    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279325    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279327    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279333    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279336    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.571479 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279339    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279341    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279344    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279349    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279352    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279354    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279357    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279360    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279362    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279365    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279378    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279381    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279385    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279388    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279390    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279393    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279396    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279398    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279401    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.571851 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279404    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279407    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279409    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279412    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279415    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279417    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279420    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279422    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279424    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279427    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279429    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279432    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279434    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279437    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279439    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279442    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279444    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279447    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279449    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.572441 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279452    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279454    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279457    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279459    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279463    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279466    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279469    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279472    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279474    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279477    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279480    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279484    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279488    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279491    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279494    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279498    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279502    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279506    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279509    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.572925 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279512    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279516    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279520    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279523    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279526    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279528    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279531    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279534    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279537    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279540    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279543    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279545    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279548    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279552    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279555    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279557    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279559    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279564    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279566    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279570    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.573394 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279573    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279575    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279578    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279581    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279583    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279586    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279588    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279590    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279594    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279596    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279599    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279602    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.279606    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279733    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279738    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:53:02.573934 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279741    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279744    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279747    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279750    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279752    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279755    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279757    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279760    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279762    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279765    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279767    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279770    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279773    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279775    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279778    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279782    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279785    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279790    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279793    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:53:02.574299 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279795    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279798    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279800    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279803    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279805    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279808    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279810    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279813    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279816    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279818    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279821    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279823    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279826    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279828    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279831    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279834    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279836    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279839    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279841    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279844    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:53:02.574805 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279846    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279848    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279851    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279853    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279856    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279859    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279874    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279877    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279882    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279887    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279891    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279893    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279896    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279898    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279901    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279903    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279906    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279908    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279911    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:53:02.575443 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279913    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279919    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279921    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279924    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279926    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279929    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279931    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279934    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279936    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279939    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279941    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279944    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279946    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279949    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279951    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279954    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279956    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279959    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279961    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:53:02.576091 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279964    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279967    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279971    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279973    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279976    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279978    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279981    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279983    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279986    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279988    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279991    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279994    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.279997    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280000    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280004    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280006    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280010    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280012    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:53:02.576581 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:02.280015    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.280019    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.280109    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.281973    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.282893    2563 server.go:1019] "Starting client certificate rotation"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.282991    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.283026    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.286413    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.287506    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.296671    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.299348    2563 log.go:25] "Validated CRI v1 image API"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.300169    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:53:02.577035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.302506    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 953a2972-020e-4685-9f27-86124cf31eae:/dev/nvme0n1p3 e0672a35-1f54-47bf-9b49-1f112faa5d97:/dev/nvme0n1p4]
Sep 09 20:53:02.577340 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.302521    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.308168    2563 manager.go:216] Machine: {Timestamp:2026-09-09 20:53:02.306925497 +0000 UTC m=+0.329092022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098290 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2138e2b95c504a47fae3e1c2ea1025 SystemUUID:ec2138e2-b95c-504a-47fa-e3e1c2ea1025 BootID:3393501d-affe-4fa6-98f3-671ef159e89d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:78:65:b1:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:78:65:b1:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:31:ef:fe:2c:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.308260    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.308355    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.310935    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.310955    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.312130    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.312140    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:53:02.577373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.312147    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.312680    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.313406    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317139    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317238    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317264    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317278    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317292    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317303    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317311    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317593    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.317608    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.318003    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.319616    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.322017    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.322054    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323718    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323741    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323750    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323759    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323773    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323782    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323802    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323811    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323820    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323829    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323852    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.323876    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.324902    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.325002    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.329233    2563 server.go:1268] "Started kubelet"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.329765    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.329813    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-22.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.329958    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.329989    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.332196    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.332236    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.333480    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:53:02.577600 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.333899    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.334480    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-22.ec2.internal.18d3c25d26759259  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-22.ec2.internal,UID:ip-10-0-138-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-22.ec2.internal,},FirstTimestamp:2026-09-09 20:53:02.329205337 +0000 UTC m=+0.351371863,LastTimestamp:2026-09-09 20:53:02.329205337 +0000 UTC m=+0.351371863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-22.ec2.internal,}"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.338017    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.342120    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.342552    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.342649    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.343129    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.343129    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.343148    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.343224    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.343232    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347244    2563 factory.go:153] Registering CRI-O factory
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347260    2563 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347310    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347320    2563 factory.go:55] Registering systemd factory
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347327    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347349    2563 factory.go:103] Registering Raw factory
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347362    2563 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.347748    2563 manager.go:349] Starting recovery of all containers
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.351145    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mqqp"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.356785    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.357296    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.364883    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mqqp"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.366321    2563 manager.go:354] Recovery completed
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.371672    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.373641    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.373664    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.373673    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374182    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374193    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374202    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374210    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:53:02.578557 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374779    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374800    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.374809    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.376423    2563 policy_none.go:49] "None policy: Start"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.377457    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.377473    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.379532    2563 policy_none.go:47] "Start"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.417433    2563 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.417474    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.417485    2563 server.go:87] "Starting device plugin registration server"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.417764    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.417775    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.417901    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.418009    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.418023    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.418670    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.418710    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.437267    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.438428    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.438473    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.438498    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.438577    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.440766    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.518202    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.519019    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.519044    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.519056    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.519077    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.528123    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.528142    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-22.ec2.internal\": node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.538842    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"]
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.539037    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.540375    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.540392    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.540401    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543052    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543226    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.579546 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543702    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543727    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543736    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543702    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543807    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.543820    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.544416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.544446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.544483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31ec0601184593e40e8b698041193b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-22.ec2.internal\" (UID: \"3e31ec0601184593e40e8b698041193b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.546459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.546483    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.547110    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.547140    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.547153    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.554947    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.572434    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-22.ec2.internal\" not found" node="ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.580589 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.578928    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-22.ec2.internal\" not found" node="ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644662 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31ec0601184593e40e8b698041193b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-22.ec2.internal\" (UID: \"3e31ec0601184593e40e8b698041193b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e31ec0601184593e40e8b698041193b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-22.ec2.internal\" (UID: \"3e31ec0601184593e40e8b698041193b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.644840 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.644742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b09faaef512ee8dcd252af95c9206a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal\" (UID: \"41b09faaef512ee8dcd252af95c9206a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.655490 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.655472    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.756059 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.756033    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.856792 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.856730    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:02.874903 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.874887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.882937 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:02.882915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:02.957111 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:02.957079    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:03.057601 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:03.057571    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.427984 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.427966    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:04.428973 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.428957    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.529382 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.529357    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.553535 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.553516    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:53:04.553661 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.553640    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:53:04.553698 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.553668    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:53:04.630216 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.630193    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.641204 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.641175    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:48:03 +0000 UTC" deadline="2028-05-20 06:14:45.644100848 +0000 UTC"
Sep 09 20:53:04.641204 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:04.641199    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14841h21m41.002904509s"
Sep 09 20:53:04.730314 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.730253    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.831160 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.831139    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:04.931682 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:04.931657    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:05.001179 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.001151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e31ec0601184593e40e8b698041193b.slice/crio-d22bad3f3afc4ea1c0993ea40f918e16fa5bdbb33c64e9985b710fda38e00c01 WatchSource:0}: Error finding container d22bad3f3afc4ea1c0993ea40f918e16fa5bdbb33c64e9985b710fda38e00c01: Status 404 returned error can't find the container with id d22bad3f3afc4ea1c0993ea40f918e16fa5bdbb33c64e9985b710fda38e00c01
Sep 09 20:53:05.005151 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.005137    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:53:05.009776 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.009755    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b09faaef512ee8dcd252af95c9206a.slice/crio-7e6f88a9384f72c2dec89182aae08a25adf4f74c1af197cd6fca634a575c8fa9 WatchSource:0}: Error finding container 7e6f88a9384f72c2dec89182aae08a25adf4f74c1af197cd6fca634a575c8fa9: Status 404 returned error can't find the container with id 7e6f88a9384f72c2dec89182aae08a25adf4f74c1af197cd6fca634a575c8fa9
Sep 09 20:53:05.031998 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.031978    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:05.053285 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.053260    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:05.132347 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.132322    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-22.ec2.internal\" not found"
Sep 09 20:53:05.144786 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.144160    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:05.213969 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.213951    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:05.229245 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.229229    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:53:05.230326 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.230313    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal"
Sep 09 20:53:05.246972 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.246926    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:53:05.573262 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.573035    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:05.589890 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.589693    2563 apiserver.go:52] "Watching apiserver"
Sep 09 20:53:05.598436 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.598144    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:53:05.600708 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.600685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-rd976","openshift-network-diagnostics/network-check-target-dj9pr","openshift-network-operator/iptables-alerter-zmqh6","openshift-ovn-kubernetes/ovnkube-node-88qn5","kube-system/global-pull-secret-syncer-bbl9s","kube-system/konnectivity-agent-f6scw","kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch","openshift-cluster-node-tuning-operator/tuned-z65tv","openshift-dns/node-resolver-86nnz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal","openshift-multus/multus-4rgv9","openshift-image-registry/node-ca-g477p","openshift-multus/multus-additional-cni-plugins-mg6s9"]
Sep 09 20:53:05.603357 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.603336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.605851 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.605828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.607145 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.607123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bpqbc\""
Sep 09 20:53:05.607334 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.607319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.607535 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.607519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.609328 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.609309    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.609563 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.609518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:53:05.609784 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.609769    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.610013 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.609998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zt2cb\""
Sep 09 20:53:05.610384 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.610369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.611068 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.611050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.612764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.612819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.612842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.613119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.613127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.613170    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:53:05.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.613302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvs95\""
Sep 09 20:53:05.613739 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.613679    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:53:05.614544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.614544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614339    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.614544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:53:05.614544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.614764 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64qrd\""
Sep 09 20:53:05.614764 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.614712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:53:05.615670 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.615166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.618121 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.617720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4zzvr\""
Sep 09 20:53:05.618121 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.617941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:53:05.618234 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.618147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:53:05.621521 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.621502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.622941 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.622920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-netd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.622957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30921f88-06ce-4cfd-aa6a-07542f3e82a6-host\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.623030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.622979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvw4\" (UniqueName: \"kubernetes.io/projected/30921f88-06ce-4cfd-aa6a-07542f3e82a6-kube-api-access-dxvw4\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.623030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwjn\" (UniqueName: \"kubernetes.io/projected/2cbf827c-79a3-4397-a70e-a9b750f004c3-kube-api-access-5cwjn\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.623030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-kubelet\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-netns\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-systemd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30921f88-06ce-4cfd-aa6a-07542f3e82a6-serviceca\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cbf827c-79a3-4397-a70e-a9b750f004c3-host-slash\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-ovn\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c8a7501-4a85-41d6-8cb2-b53133ab9993-hosts-file\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-systemd-units\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-node-log\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-config\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e54b4a-e9db-4412-ab99-bc597a89f378-ovn-node-metrics-cert\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cbf827c-79a3-4397-a70e-a9b750f004c3-iptables-alerter-script\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-var-lib-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-env-overrides\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83667412-d649-477f-b65c-14f3cc0fc2dc-agent-certs\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-log-socket\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8a7501-4a85-41d6-8cb2-b53133ab9993-tmp-dir\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-script-lib\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-dbus\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83667412-d649-477f-b65c-14f3cc0fc2dc-konnectivity-ca\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnk4w\" (UniqueName: \"kubernetes.io/projected/8c8a7501-4a85-41d6-8cb2-b53133ab9993-kube-api-access-rnk4w\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-slash\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.623810 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqwz\" (UniqueName: \"kubernetes.io/projected/90e54b4a-e9db-4412-ab99-bc597a89f378-kube-api-access-vlqwz\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.624458 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-etc-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.624458 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.624458 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-kubelet-config\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.624458 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.623774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-bin\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.625313 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.625292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.625590 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.625530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.625746 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.625727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.629338 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.629312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:05.629425 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.629398    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:05.630328 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.630295    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8bnxn\""
Sep 09 20:53:05.630790 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.630768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:53:05.631091 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.631071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wx6b\""
Sep 09 20:53:05.631341 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.631322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.635001 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.632575    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.635001 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.632631    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:53:05.635001 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.632984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:53:05.635001 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.634890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.635001 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.634965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:05.635954 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.635869    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:53:05.637162 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.637143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.639647 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.639629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.639736 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.639719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfkpq\""
Sep 09 20:53:05.640177 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.640157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:53:05.640255 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.640193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:53:05.640505 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.640461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:53:05.642654 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.642635    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:53:05.642911 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.642866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rpp6z\""
Sep 09 20:53:05.661324 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.661304    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zw7gs"
Sep 09 20:53:05.672707 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.672686    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zw7gs"
Sep 09 20:53:05.714855 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.714797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal" event={"ID":"41b09faaef512ee8dcd252af95c9206a","Type":"ContainerStarted","Data":"7e6f88a9384f72c2dec89182aae08a25adf4f74c1af197cd6fca634a575c8fa9"}
Sep 09 20:53:05.714952 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.714941    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:53:05.716420 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.716397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal" event={"ID":"3e31ec0601184593e40e8b698041193b","Type":"ContainerStarted","Data":"d22bad3f3afc4ea1c0993ea40f918e16fa5bdbb33c64e9985b710fda38e00c01"}
Sep 09 20:53:05.723978 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.723951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cx4\" (UniqueName: \"kubernetes.io/projected/57c0fccf-d75d-456a-8d40-48c4492aafde-kube-api-access-72cx4\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724073 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.723999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c8a7501-4a85-41d6-8cb2-b53133ab9993-hosts-file\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.724073 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-systemd-units\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724073 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-sys\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.724151    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c8a7501-4a85-41d6-8cb2-b53133ab9993-hosts-file\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-conf-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-systemd-units\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724217 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.724209    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.224189949 +0000 UTC m=+2.975734842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-var-lib-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cnibin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-netns\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-var-lib-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-kubernetes\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2995p\" (UniqueName: \"kubernetes.io/projected/76b32bd2-0309-478f-bc8f-6ffd579ea13f-kube-api-access-2995p\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-lib-modules\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-conf\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-device-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.724525 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-slash\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqwz\" (UniqueName: \"kubernetes.io/projected/90e54b4a-e9db-4412-ab99-bc597a89f378-kube-api-access-vlqwz\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-systemd\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-tuned\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-slash\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-system-cni-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-bin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-multus-certs\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-netd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvw4\" (UniqueName: \"kubernetes.io/projected/30921f88-06ce-4cfd-aa6a-07542f3e82a6-kube-api-access-dxvw4\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-kubelet\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-systemd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-ovn\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-var-lib-kubelet\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.724932 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-netd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-system-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-systemd\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-daemon-config\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-kubelet\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30921f88-06ce-4cfd-aa6a-07542f3e82a6-serviceca\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-ovn\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb422\" (UniqueName: \"kubernetes.io/projected/7bf54879-0091-4062-a4b2-22ba7cbfacc3-kube-api-access-jb422\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.724998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-sys-fs\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-cnibin\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-binary-copy\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-k8s-cni-cncf-io\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-node-log\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-config\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e54b4a-e9db-4412-ab99-bc597a89f378-ovn-node-metrics-cert\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-node-log\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-kubelet\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-etc-kubernetes\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30921f88-06ce-4cfd-aa6a-07542f3e82a6-serviceca\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cbf827c-79a3-4397-a70e-a9b750f004c3-iptables-alerter-script\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-env-overrides\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83667412-d649-477f-b65c-14f3cc0fc2dc-agent-certs\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysconfig\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-run\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-registration-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.725751 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-log-socket\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-modprobe-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dvp\" (UniqueName: \"kubernetes.io/projected/faa80f37-df69-406b-88f5-f74e35d7ac63-kube-api-access-b2dvp\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-multus\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-tmp\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-socket-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8a7501-4a85-41d6-8cb2-b53133ab9993-tmp-dir\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-script-lib\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-dbus\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725827    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83667412-d649-477f-b65c-14f3cc0fc2dc-konnectivity-ca\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.726167 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnk4w\" (UniqueName: \"kubernetes.io/projected/8c8a7501-4a85-41d6-8cb2-b53133ab9993-kube-api-access-rnk4w\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-hostroot\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2cbf827c-79a3-4397-a70e-a9b750f004c3-iptables-alerter-script\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-etc-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.725982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-kubelet-config\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-host\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqhf\" (UniqueName: \"kubernetes.io/projected/ca605411-f813-4395-9466-8919e71d9ac2-kube-api-access-dmqhf\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-bin\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30921f88-06ce-4cfd-aa6a-07542f3e82a6-host\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwjn\" (UniqueName: \"kubernetes.io/projected/2cbf827c-79a3-4397-a70e-a9b750f004c3-kube-api-access-5cwjn\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-netns\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.726785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cni-binary-copy\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-socket-dir-parent\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cbf827c-79a3-4397-a70e-a9b750f004c3-host-slash\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cbf827c-79a3-4397-a70e-a9b750f004c3-host-slash\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-config\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-env-overrides\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30921f88-06ce-4cfd-aa6a-07542f3e82a6-host\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-etc-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83667412-d649-477f-b65c-14f3cc0fc2dc-konnectivity-ca\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-log-socket\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.726989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-run-netns\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-run-openvswitch\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90e54b4a-e9db-4412-ab99-bc597a89f378-host-cni-bin\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-kubelet-config\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.727342 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c8a7501-4a85-41d6-8cb2-b53133ab9993-tmp-dir\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.728096 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/094fead8-5464-41e3-ab93-c0df81d66b51-dbus\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:05.728096 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.727554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90e54b4a-e9db-4412-ab99-bc597a89f378-ovnkube-script-lib\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.729870 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.729846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e54b4a-e9db-4412-ab99-bc597a89f378-ovn-node-metrics-cert\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.730817 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.730795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83667412-d649-477f-b65c-14f3cc0fc2dc-agent-certs\") pod \"konnectivity-agent-f6scw\" (UID: \"83667412-d649-477f-b65c-14f3cc0fc2dc\") " pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.738592 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.738571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvw4\" (UniqueName: \"kubernetes.io/projected/30921f88-06ce-4cfd-aa6a-07542f3e82a6-kube-api-access-dxvw4\") pod \"node-ca-g477p\" (UID: \"30921f88-06ce-4cfd-aa6a-07542f3e82a6\") " pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.742015 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.741965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqwz\" (UniqueName: \"kubernetes.io/projected/90e54b4a-e9db-4412-ab99-bc597a89f378-kube-api-access-vlqwz\") pod \"ovnkube-node-88qn5\" (UID: \"90e54b4a-e9db-4412-ab99-bc597a89f378\") " pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.746215 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.746193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwjn\" (UniqueName: \"kubernetes.io/projected/2cbf827c-79a3-4397-a70e-a9b750f004c3-kube-api-access-5cwjn\") pod \"iptables-alerter-zmqh6\" (UID: \"2cbf827c-79a3-4397-a70e-a9b750f004c3\") " pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.748059 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.748038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnk4w\" (UniqueName: \"kubernetes.io/projected/8c8a7501-4a85-41d6-8cb2-b53133ab9993-kube-api-access-rnk4w\") pod \"node-resolver-86nnz\" (UID: \"8c8a7501-4a85-41d6-8cb2-b53133ab9993\") " pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.826745 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cni-binary-copy\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826745 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-socket-dir-parent\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826745 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cx4\" (UniqueName: \"kubernetes.io/projected/57c0fccf-d75d-456a-8d40-48c4492aafde-kube-api-access-72cx4\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-sys\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-conf-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cnibin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-netns\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-kubernetes\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2995p\" (UniqueName: \"kubernetes.io/projected/76b32bd2-0309-478f-bc8f-6ffd579ea13f-kube-api-access-2995p\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.826935 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-lib-modules\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.826977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-conf\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-device-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-systemd\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-tuned\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-system-cni-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-bin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-multus-certs\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-var-lib-kubelet\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-system-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-daemon-config\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb422\" (UniqueName: \"kubernetes.io/projected/7bf54879-0091-4062-a4b2-22ba7cbfacc3-kube-api-access-jb422\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.827193 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-sys-fs\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-cnibin\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-binary-copy\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-k8s-cni-cncf-io\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-kubelet\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-etc-kubernetes\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysconfig\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-run\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-registration-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-modprobe-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.827807 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dvp\" (UniqueName: \"kubernetes.io/projected/faa80f37-df69-406b-88f5-f74e35d7ac63-kube-api-access-b2dvp\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-multus\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-tmp\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-multus-certs\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-socket-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-hostroot\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-host\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqhf\" (UniqueName: \"kubernetes.io/projected/ca605411-f813-4395-9466-8919e71d9ac2-kube-api-access-dmqhf\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-system-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-cnibin\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-sys-fs\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.828533 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-binary-copy\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cni-binary-copy\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-run\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-socket-dir-parent\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-registration-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-multus\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.827751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-socket-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.828904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-var-lib-kubelet\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-sys\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.829168 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-conf-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-hostroot\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-host\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-device-dir\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-conf\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-systemd\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-cnibin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-netns\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-kubernetes\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysctl-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-kubelet\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/faa80f37-df69-406b-88f5-f74e35d7ac63-system-cni-dir\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.829710 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-var-lib-cni-bin\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-daemon-config\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.829881    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca605411-f813-4395-9466-8919e71d9ac2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.829984    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.329966342 +0000 UTC m=+3.081511230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-sysconfig\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.830052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-multus-cni-dir\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.830086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-host-run-k8s-cni-cncf-io\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.830088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-modprobe-d\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.830110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/faa80f37-df69-406b-88f5-f74e35d7ac63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.830312 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.830215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57c0fccf-d75d-456a-8d40-48c4492aafde-lib-modules\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.830780 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.829882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76b32bd2-0309-478f-bc8f-6ffd579ea13f-etc-kubernetes\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.832787 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.832714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-tmp\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.832896 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.832819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57c0fccf-d75d-456a-8d40-48c4492aafde-etc-tuned\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.842306 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.842282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dvp\" (UniqueName: \"kubernetes.io/projected/faa80f37-df69-406b-88f5-f74e35d7ac63-kube-api-access-b2dvp\") pod \"multus-additional-cni-plugins-mg6s9\" (UID: \"faa80f37-df69-406b-88f5-f74e35d7ac63\") " pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.847162 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.847141    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:05.847261 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.847168    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:05.847261 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.847181    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:05.847261 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:05.847248    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.347230785 +0000 UTC m=+3.098775691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:05.850896 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.850869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cx4\" (UniqueName: \"kubernetes.io/projected/57c0fccf-d75d-456a-8d40-48c4492aafde-kube-api-access-72cx4\") pod \"tuned-z65tv\" (UID: \"57c0fccf-d75d-456a-8d40-48c4492aafde\") " pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.850971 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.850925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb422\" (UniqueName: \"kubernetes.io/projected/7bf54879-0091-4062-a4b2-22ba7cbfacc3-kube-api-access-jb422\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:05.852833 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.852771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqhf\" (UniqueName: \"kubernetes.io/projected/ca605411-f813-4395-9466-8919e71d9ac2-kube-api-access-dmqhf\") pod \"aws-ebs-csi-driver-node-dj8ch\" (UID: \"ca605411-f813-4395-9466-8919e71d9ac2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.853906 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.853881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2995p\" (UniqueName: \"kubernetes.io/projected/76b32bd2-0309-478f-bc8f-6ffd579ea13f-kube-api-access-2995p\") pod \"multus-4rgv9\" (UID: \"76b32bd2-0309-478f-bc8f-6ffd579ea13f\") " pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:05.922464 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.922426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86nnz"
Sep 09 20:53:05.930995 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.930970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g477p"
Sep 09 20:53:05.937056 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.937031    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8a7501_4a85_41d6_8cb2_b53133ab9993.slice/crio-458243c1704e7848fe042f23529513cc46368d4d98c5f8b4d4bcf564dc4ace51 WatchSource:0}: Error finding container 458243c1704e7848fe042f23529513cc46368d4d98c5f8b4d4bcf564dc4ace51: Status 404 returned error can't find the container with id 458243c1704e7848fe042f23529513cc46368d4d98c5f8b4d4bcf564dc4ace51
Sep 09 20:53:05.940827 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.940802    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30921f88_06ce_4cfd_aa6a_07542f3e82a6.slice/crio-d2d7d8110f2cac2215084e79776c3c612267fa142401156a8661875c6579c3bf WatchSource:0}: Error finding container d2d7d8110f2cac2215084e79776c3c612267fa142401156a8661875c6579c3bf: Status 404 returned error can't find the container with id d2d7d8110f2cac2215084e79776c3c612267fa142401156a8661875c6579c3bf
Sep 09 20:53:05.945299 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.945149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:05.953693 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.953672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zmqh6"
Sep 09 20:53:05.955057 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.955036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e54b4a_e9db_4412_ab99_bc597a89f378.slice/crio-9ba31e04479889dd54c5e0ac209e9d65a79b87e7c31a92f270c62f8cd8ec5885 WatchSource:0}: Error finding container 9ba31e04479889dd54c5e0ac209e9d65a79b87e7c31a92f270c62f8cd8ec5885: Status 404 returned error can't find the container with id 9ba31e04479889dd54c5e0ac209e9d65a79b87e7c31a92f270c62f8cd8ec5885
Sep 09 20:53:05.960327 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.960302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:05.962818 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.962776    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbf827c_79a3_4397_a70e_a9b750f004c3.slice/crio-b90e13744e8507182f4a741e3cedca60f68594780a1ac4406cc7c6a17e5e56bd WatchSource:0}: Error finding container b90e13744e8507182f4a741e3cedca60f68594780a1ac4406cc7c6a17e5e56bd: Status 404 returned error can't find the container with id b90e13744e8507182f4a741e3cedca60f68594780a1ac4406cc7c6a17e5e56bd
Sep 09 20:53:05.968648 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:05.968624    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83667412_d649_477f_b65c_14f3cc0fc2dc.slice/crio-77f81d1ae1c22a277dc1738ca5aa77f7fbb75938f1ee3efda712572073246713 WatchSource:0}: Error finding container 77f81d1ae1c22a277dc1738ca5aa77f7fbb75938f1ee3efda712572073246713: Status 404 returned error can't find the container with id 77f81d1ae1c22a277dc1738ca5aa77f7fbb75938f1ee3efda712572073246713
Sep 09 20:53:05.969309 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.969292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z65tv"
Sep 09 20:53:05.978157 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.978138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mg6s9"
Sep 09 20:53:05.986499 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.986448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch"
Sep 09 20:53:05.992917 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:05.992897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4rgv9"
Sep 09 20:53:06.230792 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.230715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:06.230940 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.230876    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:06.231054 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.230948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.230928588 +0000 UTC m=+3.982473476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:06.331762 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.331724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:06.331923 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.331831    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:06.331923 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.331883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.331870423 +0000 UTC m=+4.083415310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:06.405036 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.405006    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:53:06.432743 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.432717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:06.432914 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.432894    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:06.432967 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.432921    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:06.432967 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.432931    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:06.433025 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.432980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.432967371 +0000 UTC m=+4.184512258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:06.675734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.675647    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:48:05 +0000 UTC" deadline="2028-03-11 09:48:19.156352074 +0000 UTC"
Sep 09 20:53:06.675734 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.675682    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13164h55m12.480674832s"
Sep 09 20:53:06.710105 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.710070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:06.710245 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:06.710204    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:06.719375 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.719344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6scw" event={"ID":"83667412-d649-477f-b65c-14f3cc0fc2dc","Type":"ContainerStarted","Data":"77f81d1ae1c22a277dc1738ca5aa77f7fbb75938f1ee3efda712572073246713"}
Sep 09 20:53:06.720559 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.720509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zmqh6" event={"ID":"2cbf827c-79a3-4397-a70e-a9b750f004c3","Type":"ContainerStarted","Data":"b90e13744e8507182f4a741e3cedca60f68594780a1ac4406cc7c6a17e5e56bd"}
Sep 09 20:53:06.721690 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.721663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"9ba31e04479889dd54c5e0ac209e9d65a79b87e7c31a92f270c62f8cd8ec5885"}
Sep 09 20:53:06.722860 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.722840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g477p" event={"ID":"30921f88-06ce-4cfd-aa6a-07542f3e82a6","Type":"ContainerStarted","Data":"d2d7d8110f2cac2215084e79776c3c612267fa142401156a8661875c6579c3bf"}
Sep 09 20:53:06.723896 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:06.723866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86nnz" event={"ID":"8c8a7501-4a85-41d6-8cb2-b53133ab9993","Type":"ContainerStarted","Data":"458243c1704e7848fe042f23529513cc46368d4d98c5f8b4d4bcf564dc4ace51"}
Sep 09 20:53:06.963149 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:06.963062    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca605411_f813_4395_9466_8919e71d9ac2.slice/crio-8ed6dfa0c656944c870ed1abbfb308842de9adb97f2f408dcc3545468b537ed6 WatchSource:0}: Error finding container 8ed6dfa0c656944c870ed1abbfb308842de9adb97f2f408dcc3545468b537ed6: Status 404 returned error can't find the container with id 8ed6dfa0c656944c870ed1abbfb308842de9adb97f2f408dcc3545468b537ed6
Sep 09 20:53:06.964710 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:06.964474    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa80f37_df69_406b_88f5_f74e35d7ac63.slice/crio-1f8d524d26e9124aa01d43b8c0020d0189ab2eee119fa5f2a61ba2aa345ede49 WatchSource:0}: Error finding container 1f8d524d26e9124aa01d43b8c0020d0189ab2eee119fa5f2a61ba2aa345ede49: Status 404 returned error can't find the container with id 1f8d524d26e9124aa01d43b8c0020d0189ab2eee119fa5f2a61ba2aa345ede49
Sep 09 20:53:06.965534 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:06.965444    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b32bd2_0309_478f_bc8f_6ffd579ea13f.slice/crio-1bea543b86953de6f192ba595c5885205590bc498e6c9d21896b3f71c50c29f2 WatchSource:0}: Error finding container 1bea543b86953de6f192ba595c5885205590bc498e6c9d21896b3f71c50c29f2: Status 404 returned error can't find the container with id 1bea543b86953de6f192ba595c5885205590bc498e6c9d21896b3f71c50c29f2
Sep 09 20:53:06.966550 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:06.966509    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c0fccf_d75d_456a_8d40_48c4492aafde.slice/crio-97b02d6168b38d1c77337e5959acb12b7a779838fe9d28a71ecf6e94155892f6 WatchSource:0}: Error finding container 97b02d6168b38d1c77337e5959acb12b7a779838fe9d28a71ecf6e94155892f6: Status 404 returned error can't find the container with id 97b02d6168b38d1c77337e5959acb12b7a779838fe9d28a71ecf6e94155892f6
Sep 09 20:53:07.240654 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.240465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:07.240795 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.240632    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:07.240795 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.240734    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.240720052 +0000 UTC m=+5.992264940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:07.342422 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.341868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:07.342422 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.342048    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:07.342422 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.342106    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.342087567 +0000 UTC m=+6.093632459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:07.442771 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.442731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:07.442979 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.442945    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:07.442979 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.442965    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:07.442979 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.442977    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:07.443140 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.443036    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.44301832 +0000 UTC m=+6.194563229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:07.675887 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.675780    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:48:05 +0000 UTC" deadline="2028-04-02 11:36:56.137774889 +0000 UTC"
Sep 09 20:53:07.675887 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.675840    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13694h43m48.461939075s"
Sep 09 20:53:07.711231 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.711203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:07.712887 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.712854    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:07.713249 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.713230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:07.713356 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:07.713337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:07.730456 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.730363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rgv9" event={"ID":"76b32bd2-0309-478f-bc8f-6ffd579ea13f","Type":"ContainerStarted","Data":"1bea543b86953de6f192ba595c5885205590bc498e6c9d21896b3f71c50c29f2"}
Sep 09 20:53:07.735804 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.735761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch" event={"ID":"ca605411-f813-4395-9466-8919e71d9ac2","Type":"ContainerStarted","Data":"8ed6dfa0c656944c870ed1abbfb308842de9adb97f2f408dcc3545468b537ed6"}
Sep 09 20:53:07.738186 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.738124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z65tv" event={"ID":"57c0fccf-d75d-456a-8d40-48c4492aafde","Type":"ContainerStarted","Data":"97b02d6168b38d1c77337e5959acb12b7a779838fe9d28a71ecf6e94155892f6"}
Sep 09 20:53:07.743014 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.742954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerStarted","Data":"1f8d524d26e9124aa01d43b8c0020d0189ab2eee119fa5f2a61ba2aa345ede49"}
Sep 09 20:53:07.769725 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.769696    2563 generic.go:356] "Generic (PLEG): container finished" podID="41b09faaef512ee8dcd252af95c9206a" containerID="622f0aeea88ad424e04cb0076a0a8b4d611603fdef732ba3fa7ffbd80f4ecb4d" exitCode=0
Sep 09 20:53:07.769823 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.769775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal" event={"ID":"41b09faaef512ee8dcd252af95c9206a","Type":"ContainerDied","Data":"622f0aeea88ad424e04cb0076a0a8b4d611603fdef732ba3fa7ffbd80f4ecb4d"}
Sep 09 20:53:07.774606 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.774164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal" event={"ID":"3e31ec0601184593e40e8b698041193b","Type":"ContainerStarted","Data":"0f1d91fd7978afe0d256ff3f6d0ffc2fe225bc4de74b62a8fe17eb46f211c942"}
Sep 09 20:53:07.811831 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:07.811783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-22.ec2.internal" podStartSLOduration=2.811763823 podStartE2EDuration="2.811763823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:07.811151906 +0000 UTC m=+4.562696840" watchObservedRunningTime="2026-09-09 20:53:07.811763823 +0000 UTC m=+4.563308733"
Sep 09 20:53:08.710218 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:08.710176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:08.710895 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:08.710269    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:08.789339 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:08.789269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal" event={"ID":"41b09faaef512ee8dcd252af95c9206a","Type":"ContainerStarted","Data":"fc777ce6c06331a92a50417198af9e930c4238922d913438d4ef5325e628338d"}
Sep 09 20:53:08.813969 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:08.813907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-22.ec2.internal" podStartSLOduration=3.813884548 podStartE2EDuration="3.813884548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:08.812827599 +0000 UTC m=+5.564372509" watchObservedRunningTime="2026-09-09 20:53:08.813884548 +0000 UTC m=+5.565429460"
Sep 09 20:53:09.257234 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:09.257186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:09.257423 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.257336    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:09.257423 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.257396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.257377388 +0000 UTC m=+10.008922288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:09.357993 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:09.357954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:09.358174 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.358152    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:09.358246 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.358217    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.358198089 +0000 UTC m=+10.109742983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:09.459209 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:09.459130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:09.459408 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.459289    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:09.459408 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.459312    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:09.459408 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.459326    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:09.459408 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.459383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.45936427 +0000 UTC m=+10.210909161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:09.710673 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:09.710595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:09.711127 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.710744    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:09.711127 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:09.710782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:09.711127 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:09.710858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:10.709748 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:10.709718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:10.709940 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:10.709838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:11.709903 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:11.709656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:11.709903 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:11.709691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:11.709903 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:11.709797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:11.709903 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:11.709892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:12.709570 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:12.709500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:12.709713 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:12.709635    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:13.295513 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:13.295463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:13.295959 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.295605    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:13.295959 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.295713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.295693068 +0000 UTC m=+18.047237957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:13.396969 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:13.396340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:13.396969 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.396557    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:13.396969 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.396618    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.396598714 +0000 UTC m=+18.148143616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:13.496806 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:13.496767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:13.496989 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.496954    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:13.496989 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.496984    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:13.497097 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.496998    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:13.497097 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.497063    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.497047985 +0000 UTC m=+18.248592877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:13.711026 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:13.710956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:13.711177 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.711155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:13.711247 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:13.711104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:13.711301 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:13.711268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:14.709552 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:14.709518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:14.710109 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:14.709639    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:15.710054 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:15.710020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:15.710476 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:15.710022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:15.710476 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:15.710147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:15.710476 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:15.710244    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:16.709470 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:16.709444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:16.709631 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:16.709543    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:17.709815 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:17.709781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:17.710322 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:17.709781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:17.710322 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:17.709915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:17.710322 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:17.710013    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:18.709993 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:18.709954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:18.710401 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:18.710086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:19.710202 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:19.710166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:19.710626 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:19.710298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:19.710626 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:19.710351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:19.710626 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:19.710446    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:20.709574 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:20.709539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:20.709751 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:20.709666    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:21.351645 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:21.351607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:21.352252 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.351728    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:21.352252 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.351792    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:37.351772723 +0000 UTC m=+34.103317630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:21.452224 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:21.452184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:21.452408 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.452355    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:21.452475 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.452438    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:37.452415412 +0000 UTC m=+34.203960314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:21.552795 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:21.552750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:21.552958 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.552890    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:21.552958 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.552912    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:21.552958 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.552921    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:21.553091 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.552978    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:37.552959419 +0000 UTC m=+34.304504325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:21.710283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:21.710249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:21.710283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:21.710280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:21.710533 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.710383    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:21.710601 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:21.710526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:22.710066 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:22.710038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:22.710461 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:22.710155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:23.709958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:23.709923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:23.709958 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:23.709951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:23.710419 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:23.710041    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:23.710419 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:23.710141    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:24.709879 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.709671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:24.710028 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:24.709996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:24.822548 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.822520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6scw" event={"ID":"83667412-d649-477f-b65c-14f3cc0fc2dc","Type":"ContainerStarted","Data":"d009bab125da4b385cd8746081e421cdae80b1e0e2336ecef6a6ef21e5445337"}
Sep 09 20:53:24.824698 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.824367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86nnz" event={"ID":"8c8a7501-4a85-41d6-8cb2-b53133ab9993","Type":"ContainerStarted","Data":"938c7e7937547f814b7f3bd5085e88ba983cce94f267f3041762cc7c97cda319"}
Sep 09 20:53:24.826479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.826449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4rgv9" event={"ID":"76b32bd2-0309-478f-bc8f-6ffd579ea13f","Type":"ContainerStarted","Data":"5e72734e76987cef5a772ab52c402f8fa80094447a6622e3de043cbc2a499a92"}
Sep 09 20:53:24.828266 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.828244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch" event={"ID":"ca605411-f813-4395-9466-8919e71d9ac2","Type":"ContainerStarted","Data":"e35ae6047169c9a7a9b7ab68cdf176c6fbb3b4c5436529909e935aa4bae5e31b"}
Sep 09 20:53:24.831105 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.831082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"534161a6694b2fbf08c34c493813b2f2159f277bf2b09d3badffa1b5c21cf2bc"}
Sep 09 20:53:24.831199 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.831143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"3cfd1f5a5c35a0e2cbfcf8f73ce6eed5df3401a75049bcb04c2f53d0ad013d1d"}
Sep 09 20:53:24.831199 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.831177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"9d2fd61b635ccd56fdaae1a939946c89af7f75c479cd423c64b65e1269cb4630"}
Sep 09 20:53:24.831294 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.831221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"bb81018cdd12bab2077a7e17b304d055b3abb39b11927f1176300cb26789637e"}
Sep 09 20:53:24.832603 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.832579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g477p" event={"ID":"30921f88-06ce-4cfd-aa6a-07542f3e82a6","Type":"ContainerStarted","Data":"cab266cdc36e86bc859d8d2f3dc13bcf1637f7fc2dfe2a251ba4bd3d06cab0f0"}
Sep 09 20:53:24.834080 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.834054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z65tv" event={"ID":"57c0fccf-d75d-456a-8d40-48c4492aafde","Type":"ContainerStarted","Data":"8b795bde292ae2e346ef1426b76323d57d35de8e59fbbbe18e572a583dc90af1"}
Sep 09 20:53:24.835826 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.835806    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="9977adc60f6fa845e32f632c7bf10803aa910318acfc13448b3efa6aff655f4e" exitCode=0
Sep 09 20:53:24.835893 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.835843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"9977adc60f6fa845e32f632c7bf10803aa910318acfc13448b3efa6aff655f4e"}
Sep 09 20:53:24.846795 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.846745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f6scw" podStartSLOduration=3.428949016 podStartE2EDuration="21.846728169s" totalImagesPullingTime="18.417779153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.84656384 +0000 UTC m=+21.598108749" watchObservedRunningTime="2026-09-09 20:53:24.846728169 +0000 UTC m=+21.598273079"
Sep 09 20:53:24.871438 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.871397    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-86nnz" podStartSLOduration=3.404477101 podStartE2EDuration="21.871385515s" totalImagesPullingTime="18.466908414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.871255136 +0000 UTC m=+21.622800047" watchObservedRunningTime="2026-09-09 20:53:24.871385515 +0000 UTC m=+21.622930423"
Sep 09 20:53:24.892220 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.892180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g477p" podStartSLOduration=7.335805605 podStartE2EDuration="21.892166914s" totalImagesPullingTime="14.556361309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.891970755 +0000 UTC m=+21.643515664" watchObservedRunningTime="2026-09-09 20:53:24.892166914 +0000 UTC m=+21.643711823"
Sep 09 20:53:24.920934 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.920894    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z65tv" podStartSLOduration=4.469644992 podStartE2EDuration="21.92087987s" totalImagesPullingTime="17.451234878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.92061808 +0000 UTC m=+21.672163009" watchObservedRunningTime="2026-09-09 20:53:24.92087987 +0000 UTC m=+21.672424778"
Sep 09 20:53:24.980395 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:24.980349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4rgv9" podStartSLOduration=4.441112592 podStartE2EDuration="21.980335764s" totalImagesPullingTime="17.539223172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.979711249 +0000 UTC m=+21.731256157" watchObservedRunningTime="2026-09-09 20:53:24.980335764 +0000 UTC m=+21.731880672"
Sep 09 20:53:25.710266 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.710189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:25.710266 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.710227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:25.710498 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:25.710347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:25.710563 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:25.710475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:25.840550 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.840514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zmqh6" event={"ID":"2cbf827c-79a3-4397-a70e-a9b750f004c3","Type":"ContainerStarted","Data":"647ba9294c2d2a99646ce234179171c763b356c682dae7e9c5b3e92047d22400"}
Sep 09 20:53:25.844177 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.844149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"698804b743461eb651a7031e5c97d1e1110b111b4eae5f09ea5838171480e500"}
Sep 09 20:53:25.844279 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.844234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"168682cef2d426e4b7c637afba85f2c78f32b59f31959b0e05dd634e613f68d7"}
Sep 09 20:53:25.888722 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:25.888696    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:53:26.699940 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:26.699817    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:53:25.888719844Z","UUID":"d42f8e8c-00f8-42e4-bb7c-64b8e9b340c4","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:53:26.701597 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:26.701570    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:53:26.701715 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:26.701615    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:53:26.709311 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:26.709279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:26.709442 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:26.709404    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:26.849047 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:26.849015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch" event={"ID":"ca605411-f813-4395-9466-8919e71d9ac2","Type":"ContainerStarted","Data":"b09955a457220b6c07a06ecba2658e231c5d761f74c7a819c1fe86e1c0822bd8"}
Sep 09 20:53:27.710348 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.710315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:27.710650 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.710315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:27.710650 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:27.710439    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:27.710650 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:27.710530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:27.730251 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.730219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:27.731033 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.731011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:27.749711 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.749670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zmqh6" podStartSLOduration=6.292418016 podStartE2EDuration="24.749658073s" totalImagesPullingTime="18.457240057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:25.863958004 +0000 UTC m=+22.615502934" watchObservedRunningTime="2026-09-09 20:53:27.749658073 +0000 UTC m=+24.501203026"
Sep 09 20:53:27.853927 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.853884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch" event={"ID":"ca605411-f813-4395-9466-8919e71d9ac2","Type":"ContainerStarted","Data":"da9fc2833e07132132d81dbf89b7b522434d79a430396191062594c08e9ac863"}
Sep 09 20:53:27.858217 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.858184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"fc9050e6c71cfd5f07bab610d81a4cec8e2db66cdd70c316e3ee175ce654da19"}
Sep 09 20:53:27.858425 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.858392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:27.858861 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.858839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f6scw"
Sep 09 20:53:27.874724 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:27.874685    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dj8ch" podStartSLOduration=5.388292309 podStartE2EDuration="24.87467278s" totalImagesPullingTime="19.486380471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:27.874644274 +0000 UTC m=+24.626189182" watchObservedRunningTime="2026-09-09 20:53:27.87467278 +0000 UTC m=+24.626217685"
Sep 09 20:53:28.710118 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:28.709938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:28.710253 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:28.710185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:29.709968 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:29.709934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:29.710452 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:29.710046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:29.710452 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:29.710110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:29.710452 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:29.710227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:29.864143 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:29.864109    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="bdb816688df6c43071ac2e29dd43b64a59068ae39c05c696d0a1c7c167ab686a" exitCode=0
Sep 09 20:53:29.864272 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:29.864190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"bdb816688df6c43071ac2e29dd43b64a59068ae39c05c696d0a1c7c167ab686a"}
Sep 09 20:53:29.868424 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:29.868228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" event={"ID":"90e54b4a-e9db-4412-ab99-bc597a89f378","Type":"ContainerStarted","Data":"d8e4bb46e76a1bcbbbf0901a6529e7b9af9e05b2c7313aab65362ca88b7d0baf"}
Sep 09 20:53:30.709340 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.709309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:30.709509 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:30.709432    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:30.872770 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.872735    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="54e4e63c6ab3e933f4c7a42907bd38dc72916b7aa863bf2ad8c770c740bcebf0" exitCode=0
Sep 09 20:53:30.873266 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.872798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"54e4e63c6ab3e933f4c7a42907bd38dc72916b7aa863bf2ad8c770c740bcebf0"}
Sep 09 20:53:30.873266 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.872892    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:53:30.873939 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.873905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:30.874044 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.873997    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:30.891055 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.891034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:30.891625 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.891608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:30.905992 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:30.905950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5" podStartSLOduration=9.412330333 podStartE2EDuration="27.905936877s" totalImagesPullingTime="18.493606544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:29.940747047 +0000 UTC m=+26.692291966" watchObservedRunningTime="2026-09-09 20:53:30.905936877 +0000 UTC m=+27.657481785"
Sep 09 20:53:31.132931 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.132850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbl9s"]
Sep 09 20:53:31.133080 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.133006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:31.133136 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:31.133114    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:31.137283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.137258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dj9pr"]
Sep 09 20:53:31.137408 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.137355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:31.137466 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:31.137443    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:31.138459 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.138433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rd976"]
Sep 09 20:53:31.138729 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.138542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:31.138729 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:31.138637    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:31.877464 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.877433    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="45408a2dc9432c594eb01ab457263684f1a79a2e286063656f264f95eb7cf1d4" exitCode=0
Sep 09 20:53:31.878111 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.877515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"45408a2dc9432c594eb01ab457263684f1a79a2e286063656f264f95eb7cf1d4"}
Sep 09 20:53:31.878111 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:31.877594    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:53:32.709844 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:32.709800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:32.710037 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:32.709850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:32.710037 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:32.709816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:32.710037 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:32.709925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:32.710037 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:32.709968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:32.710219 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:32.710043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:32.879708 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:32.879677    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:53:33.267058 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:33.267023    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:53:34.709438 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:34.709400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:34.709438 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:34.709430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:34.709925 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:34.709430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:34.709925 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:34.709556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:34.709925 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:34.709626    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:34.709925 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:34.709690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:36.710164 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:36.710134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:36.710646 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:36.710234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:36.710646 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:36.710243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dj9pr" podUID="063eda39-99d3-43a4-861f-55406e55f550"
Sep 09 20:53:36.710646 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:36.710366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rd976" podUID="7bf54879-0091-4062-a4b2-22ba7cbfacc3"
Sep 09 20:53:36.710646 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:36.710415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:36.710646 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:36.710510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbl9s" podUID="094fead8-5464-41e3-ab93-c0df81d66b51"
Sep 09 20:53:36.891831 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:36.891801    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="ed7005460b27c38b55e0fe850bae6af058876cdec6db6800cf7ffd083204aa6e" exitCode=0
Sep 09 20:53:36.892031 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:36.891851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"ed7005460b27c38b55e0fe850bae6af058876cdec6db6800cf7ffd083204aa6e"}
Sep 09 20:53:37.372857 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.372827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:37.373090 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.372955    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:37.373090 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.373003    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret podName:094fead8-5464-41e3-ab93-c0df81d66b51 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.372989942 +0000 UTC m=+66.124534829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret") pod "global-pull-secret-syncer-bbl9s" (UID: "094fead8-5464-41e3-ab93-c0df81d66b51") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:37.473754 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.473717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:37.473897 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.473828    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:37.473897 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.473884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.47386976 +0000 UTC m=+66.225414646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:37.543371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.543346    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-22.ec2.internal" event="NodeReady"
Sep 09 20:53:37.543544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.543469    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:53:37.574952 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.574925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:37.575063 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.575052    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:37.575114 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.575064    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:37.575114 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.575073    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cb2zg for pod openshift-network-diagnostics/network-check-target-dj9pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:37.575114 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.575112    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg podName:063eda39-99d3-43a4-861f-55406e55f550 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.575099709 +0000 UTC m=+66.326644596 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cb2zg" (UniqueName: "kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg") pod "network-check-target-dj9pr" (UID: "063eda39-99d3-43a4-861f-55406e55f550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:37.618403 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.618151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xmv6d"]
Sep 09 20:53:37.631481 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.631434    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xjgzb"]
Sep 09 20:53:37.631592 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.631574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.634230 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.634209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:53:37.634388 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.634371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:53:37.634445 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.634427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7nzq\""
Sep 09 20:53:37.634735 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.634722    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:53:37.643480 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.643461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmv6d"]
Sep 09 20:53:37.643589 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.643507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xjgzb"]
Sep 09 20:53:37.643630 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.643597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.649420 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.649402    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:53:37.649522 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.649434    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:53:37.650112 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.650098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:53:37.776587 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.776950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe439eb-c21c-4593-b642-c613c44889bd-config-volume\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.776950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qknj\" (UniqueName: \"kubernetes.io/projected/9e374074-9180-4a2a-91b4-36f06f306cfe-kube-api-access-7qknj\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.776950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe439eb-c21c-4593-b642-c613c44889bd-tmp-dir\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.776950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e374074-9180-4a2a-91b4-36f06f306cfe-cert\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.776950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.776751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qqq\" (UniqueName: \"kubernetes.io/projected/1fe439eb-c21c-4593-b642-c613c44889bd-kube-api-access-s8qqq\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.877857 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.877828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e374074-9180-4a2a-91b4-36f06f306cfe-cert\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.877857 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.877860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qqq\" (UniqueName: \"kubernetes.io/projected/1fe439eb-c21c-4593-b642-c613c44889bd-kube-api-access-s8qqq\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.878030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.877881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.878030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.877977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe439eb-c21c-4593-b642-c613c44889bd-config-volume\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.878030 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.878009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qknj\" (UniqueName: \"kubernetes.io/projected/9e374074-9180-4a2a-91b4-36f06f306cfe-kube-api-access-7qknj\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.878150 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.878032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe439eb-c21c-4593-b642-c613c44889bd-tmp-dir\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.878150 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.877988    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:37.878150 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:37.878112    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:38.378091769 +0000 UTC m=+35.129636664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:37.878387 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.878369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe439eb-c21c-4593-b642-c613c44889bd-tmp-dir\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.878636 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.878617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe439eb-c21c-4593-b642-c613c44889bd-config-volume\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.881842 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.881793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e374074-9180-4a2a-91b4-36f06f306cfe-cert\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.891617 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.891598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qqq\" (UniqueName: \"kubernetes.io/projected/1fe439eb-c21c-4593-b642-c613c44889bd-kube-api-access-s8qqq\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:37.891707 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.891691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qknj\" (UniqueName: \"kubernetes.io/projected/9e374074-9180-4a2a-91b4-36f06f306cfe-kube-api-access-7qknj\") pod \"ingress-canary-xmv6d\" (UID: \"9e374074-9180-4a2a-91b4-36f06f306cfe\") " pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:37.897499 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.897463    2563 generic.go:356] "Generic (PLEG): container finished" podID="faa80f37-df69-406b-88f5-f74e35d7ac63" containerID="5313f4cf75a166492519d5f74bcdaa3bb92885932eb6b52a12079d0573335db4" exitCode=0
Sep 09 20:53:37.897586 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.897513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerDied","Data":"5313f4cf75a166492519d5f74bcdaa3bb92885932eb6b52a12079d0573335db4"}
Sep 09 20:53:37.940590 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:37.940438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xmv6d"
Sep 09 20:53:38.255561 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.255480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xmv6d"]
Sep 09 20:53:38.258462 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:53:38.258438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e374074_9180_4a2a_91b4_36f06f306cfe.slice/crio-19ee93cd4e68cd99285199fdb4f7e8c614872a22d434a268098368fb89256eba WatchSource:0}: Error finding container 19ee93cd4e68cd99285199fdb4f7e8c614872a22d434a268098368fb89256eba: Status 404 returned error can't find the container with id 19ee93cd4e68cd99285199fdb4f7e8c614872a22d434a268098368fb89256eba
Sep 09 20:53:38.381925 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.381899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:38.382121 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:38.382045    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:38.382121 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:38.382103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:39.382087491 +0000 UTC m=+36.133632378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:38.709544 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.709513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:53:38.709870 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.709513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:53:38.709870 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.709513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:53:38.712673 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.712649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:38.712673 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.712657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:53:38.712673 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.712651    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:38.713003 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.712988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:38.713801 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.713783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:38.713903 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.713889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:53:38.904686 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.904656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" event={"ID":"faa80f37-df69-406b-88f5-f74e35d7ac63","Type":"ContainerStarted","Data":"7b8301e9fb4e5fe0d9c965c87992b4ac9c553b1f123ebdbd844f1965de11cdfd"}
Sep 09 20:53:38.906108 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:38.906077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmv6d" event={"ID":"9e374074-9180-4a2a-91b4-36f06f306cfe","Type":"ContainerStarted","Data":"19ee93cd4e68cd99285199fdb4f7e8c614872a22d434a268098368fb89256eba"}
Sep 09 20:53:39.388735 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:39.388700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:39.388910 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:39.388811    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:39.388910 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:39.388874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:41.388854501 +0000 UTC m=+38.140399406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:40.913247 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:40.913175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xmv6d" event={"ID":"9e374074-9180-4a2a-91b4-36f06f306cfe","Type":"ContainerStarted","Data":"d087218de6d799058b25c40a9c0f5c1aeb94d182f5c76edb5c0cb08bb8c66c50"}
Sep 09 20:53:40.935387 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:40.935324    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mg6s9" podStartSLOduration=10.743789468 podStartE2EDuration="37.935305565s" totalImagesPullingTime="27.191516097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:38.933982836 +0000 UTC m=+35.685527746" watchObservedRunningTime="2026-09-09 20:53:40.935305565 +0000 UTC m=+37.686850473"
Sep 09 20:53:40.935636 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:40.935611    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xmv6d" podStartSLOduration=1.42678512 podStartE2EDuration="3.935601566s" totalImagesPullingTime="2.508816446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:40.93542223 +0000 UTC m=+37.686967137" watchObservedRunningTime="2026-09-09 20:53:40.935601566 +0000 UTC m=+37.687146475"
Sep 09 20:53:41.404055 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:41.404024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:41.404201 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:41.404168    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:41.404242 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:41.404228    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:45.404212102 +0000 UTC m=+42.155756989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:45.432308 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:45.432241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:45.432722 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:45.432401    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:45.432722 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:45.432497    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:53:53.432465302 +0000 UTC m=+50.184010190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:53.484410 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:53:53.484371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:53:53.484783 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:53.484514    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:53.484783 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:53:53.484566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.484552224 +0000 UTC m=+66.236097111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:54:03.891112 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:03.891088    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88qn5"
Sep 09 20:54:08.747373 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.747335    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fckws"]
Sep 09 20:54:08.749828 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.749812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.752629 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.752604    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:54:08.752725 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.752687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:54:08.752777 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.752728    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:54:08.753835 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.753817    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:54:08.753910 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.753840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pqhgc\""
Sep 09 20:54:08.758472 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.758455    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:54:08.771551 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.771529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fckws"]
Sep 09 20:54:08.841551 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.841528    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"]
Sep 09 20:54:08.843105 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.843091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:08.847538 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.847518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-c446l\""
Sep 09 20:54:08.847635 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.847538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:54:08.847635 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.847568    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:54:08.847635 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.847605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:54:08.847859 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.847843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:54:08.860432 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.860414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"]
Sep 09 20:54:08.889274 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-snapshots\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.889371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.889371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.889371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f146f95f-c995-445f-9d54-dc3931306c22-serving-cert\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.889469 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcf5h\" (UniqueName: \"kubernetes.io/projected/f146f95f-c995-445f-9d54-dc3931306c22-kube-api-access-wcf5h\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.889469 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.889426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-tmp\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990082 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7615b056-272c-4d62-9ea5-37861f719f42-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:08.990179 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn647\" (UniqueName: \"kubernetes.io/projected/7615b056-272c-4d62-9ea5-37861f719f42-kube-api-access-nn647\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:08.990179 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-snapshots\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990179 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990321 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990321 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f146f95f-c995-445f-9d54-dc3931306c22-serving-cert\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990423 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcf5h\" (UniqueName: \"kubernetes.io/projected/f146f95f-c995-445f-9d54-dc3931306c22-kube-api-access-wcf5h\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990423 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-tmp\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990423 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:08.990680 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990727 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-tmp\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.990859 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.990837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f146f95f-c995-445f-9d54-dc3931306c22-snapshots\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.991670 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.991654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f146f95f-c995-445f-9d54-dc3931306c22-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:08.993492 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:08.993473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f146f95f-c995-445f-9d54-dc3931306c22-serving-cert\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:09.000463 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.000406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcf5h\" (UniqueName: \"kubernetes.io/projected/f146f95f-c995-445f-9d54-dc3931306c22-kube-api-access-wcf5h\") pod \"insights-operator-76bcd6ccf5-fckws\" (UID: \"f146f95f-c995-445f-9d54-dc3931306c22\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:09.058717 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.058689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws"
Sep 09 20:54:09.090916 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.090884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7615b056-272c-4d62-9ea5-37861f719f42-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.090916 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.090921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn647\" (UniqueName: \"kubernetes.io/projected/7615b056-272c-4d62-9ea5-37861f719f42-kube-api-access-nn647\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.091148 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.091074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.091227 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.091212    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:09.091287 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.091281    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:09.59126369 +0000 UTC m=+66.342808579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:09.091543 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.091522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7615b056-272c-4d62-9ea5-37861f719f42-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.101325 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.101280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn647\" (UniqueName: \"kubernetes.io/projected/7615b056-272c-4d62-9ea5-37861f719f42-kube-api-access-nn647\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.339140 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.339113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fckws"]
Sep 09 20:54:09.341376 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:09.341349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf146f95f_c995_445f_9d54_dc3931306c22.slice/crio-089a72dd20d5c3d5da979a39997998785b7718594b40f3378ce7f4278213c19b WatchSource:0}: Error finding container 089a72dd20d5c3d5da979a39997998785b7718594b40f3378ce7f4278213c19b: Status 404 returned error can't find the container with id 089a72dd20d5c3d5da979a39997998785b7718594b40f3378ce7f4278213c19b
Sep 09 20:54:09.393338 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.393313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:54:09.396771 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.396752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:54:09.406433 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.406412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/094fead8-5464-41e3-ab93-c0df81d66b51-original-pull-secret\") pod \"global-pull-secret-syncer-bbl9s\" (UID: \"094fead8-5464-41e3-ab93-c0df81d66b51\") " pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:54:09.494480 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.494456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:54:09.494594 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.494530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:09.494653 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.494637    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:54:09.494708 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.494699    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls podName:1fe439eb-c21c-4593-b642-c613c44889bd nodeName:}" failed. No retries permitted until 2026-09-09 20:54:41.494681846 +0000 UTC m=+98.246226747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls") pod "dns-default-xjgzb" (UID: "1fe439eb-c21c-4593-b642-c613c44889bd") : secret "dns-default-metrics-tls" not found
Sep 09 20:54:09.498092 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.498074    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:54:09.504827 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.504811    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:54:09.504901 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.504861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs podName:7bf54879-0091-4062-a4b2-22ba7cbfacc3 nodeName:}" failed. No retries permitted until 2026-09-09 20:55:13.504844989 +0000 UTC m=+130.256389893 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs") pod "network-metrics-daemon-rd976" (UID: "7bf54879-0091-4062-a4b2-22ba7cbfacc3") : secret "metrics-daemon-secret" not found
Sep 09 20:54:09.595378 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.595308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:09.595378 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.595356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:54:09.595551 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.595434    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:09.595714 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:09.595696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:10.595620195 +0000 UTC m=+67.347165095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:09.599038 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.599019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:54:09.609378 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.609360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:54:09.619013 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.618989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2zg\" (UniqueName: \"kubernetes.io/projected/063eda39-99d3-43a4-861f-55406e55f550-kube-api-access-cb2zg\") pod \"network-check-target-dj9pr\" (UID: \"063eda39-99d3-43a4-861f-55406e55f550\") " pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:54:09.622639 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.622625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:54:09.625792 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.625778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbl9s"
Sep 09 20:54:09.630371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.630358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:54:09.902092 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.902047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbl9s"]
Sep 09 20:54:09.903045 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.903018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dj9pr"]
Sep 09 20:54:09.905692 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:09.905665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063eda39_99d3_43a4_861f_55406e55f550.slice/crio-35e8c024997fa67510b4d98f4f706089a9d91c360d8502044f6b29c0a2805aa3 WatchSource:0}: Error finding container 35e8c024997fa67510b4d98f4f706089a9d91c360d8502044f6b29c0a2805aa3: Status 404 returned error can't find the container with id 35e8c024997fa67510b4d98f4f706089a9d91c360d8502044f6b29c0a2805aa3
Sep 09 20:54:09.906553 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:09.906533    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094fead8_5464_41e3_ab93_c0df81d66b51.slice/crio-9f2756d95a16f9aaf983c8c5b44b76cc7d36b801f92bd38755895be73ba6627c WatchSource:0}: Error finding container 9f2756d95a16f9aaf983c8c5b44b76cc7d36b801f92bd38755895be73ba6627c: Status 404 returned error can't find the container with id 9f2756d95a16f9aaf983c8c5b44b76cc7d36b801f92bd38755895be73ba6627c
Sep 09 20:54:09.982343 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.982300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dj9pr" event={"ID":"063eda39-99d3-43a4-861f-55406e55f550","Type":"ContainerStarted","Data":"35e8c024997fa67510b4d98f4f706089a9d91c360d8502044f6b29c0a2805aa3"}
Sep 09 20:54:09.983628 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.983596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws" event={"ID":"f146f95f-c995-445f-9d54-dc3931306c22","Type":"ContainerStarted","Data":"089a72dd20d5c3d5da979a39997998785b7718594b40f3378ce7f4278213c19b"}
Sep 09 20:54:09.984845 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:09.984818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbl9s" event={"ID":"094fead8-5464-41e3-ab93-c0df81d66b51","Type":"ContainerStarted","Data":"9f2756d95a16f9aaf983c8c5b44b76cc7d36b801f92bd38755895be73ba6627c"}
Sep 09 20:54:10.601542 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:10.601501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:10.601715 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:10.601630    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:10.601778 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:10.601713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:12.601690881 +0000 UTC m=+69.353235772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:11.992531 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:11.992467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws" event={"ID":"f146f95f-c995-445f-9d54-dc3931306c22","Type":"ContainerStarted","Data":"4843cc6ff4fa8fcc574ad45eab63679e16310859caf2777dafb352dd463075f6"}
Sep 09 20:54:12.031154 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.031056    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws" podStartSLOduration=2.131261893 podStartE2EDuration="4.031035069s" totalImagesPullingTime="1.899773176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:12.030816337 +0000 UTC m=+68.782361247" watchObservedRunningTime="2026-09-09 20:54:12.031035069 +0000 UTC m=+68.782579978"
Sep 09 20:54:12.569626 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.569587    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2g946"]
Sep 09 20:54:12.572669 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.572641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.576052 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.575781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:54:12.576052 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.575797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:54:12.576052 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.575842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksvhl\""
Sep 09 20:54:12.588635 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.588568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2g946"]
Sep 09 20:54:12.618074 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-crio-socket\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.618222 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.618222 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.618222 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6rq\" (UniqueName: \"kubernetes.io/projected/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-api-access-rr6rq\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.618390 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-data-volume\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.618390 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.618321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:12.618523 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.618505    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:12.618584 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.618575    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.618556447 +0000 UTC m=+73.370101340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:12.718748 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.718696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-crio-socket\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.718759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.718786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.718834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6rq\" (UniqueName: \"kubernetes.io/projected/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-api-access-rr6rq\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.718851    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.718864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-data-volume\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.718882    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.718923 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.718918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.218900436 +0000 UTC m=+69.970445324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.719314 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:12.718932    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.218926329 +0000 UTC m=+69.970471216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.719314 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.719091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-crio-socket\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.719314 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.719242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-data-volume\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:12.732972 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:12.732946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6rq\" (UniqueName: \"kubernetes.io/projected/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-api-access-rr6rq\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:13.223391 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:13.223346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:13.223391 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:13.223383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:13.223987 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:13.223465    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.223987 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:13.223551    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.223533505 +0000 UTC m=+70.975078392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.223987 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:13.223479    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:13.223987 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:13.223604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.223592328 +0000 UTC m=+70.975137215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.231587 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:14.231547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:14.231587 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:14.231589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:14.232124 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:14.231675    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.232124 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:14.231691    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.232124 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:14.231749    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.231731526 +0000 UTC m=+72.983276417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.232124 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:14.231768    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.231759536 +0000 UTC m=+72.983304423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.291922 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:14.291895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86nnz_8c8a7501-4a85-41d6-8cb2-b53133ab9993/dns-node-resolver/0.log"
Sep 09 20:54:15.003426 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.003336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbl9s" event={"ID":"094fead8-5464-41e3-ab93-c0df81d66b51","Type":"ContainerStarted","Data":"837e3126ba80b424ac0afe16610a1ee64a19ec6033ba4158c5bcdfd576961f62"}
Sep 09 20:54:15.004861 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.004831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dj9pr" event={"ID":"063eda39-99d3-43a4-861f-55406e55f550","Type":"ContainerStarted","Data":"527797f82b85444d4c78c29e5744980816d9c477321a105572faca55099e99d6"}
Sep 09 20:54:15.004990 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.004975    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:54:15.020804 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.020763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bbl9s" podStartSLOduration=67.270524752 podStartE2EDuration="1m12.020747876s" totalImagesPullingTime="4.750223124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:15.019896217 +0000 UTC m=+71.771441127" watchObservedRunningTime="2026-09-09 20:54:15.020747876 +0000 UTC m=+71.772292783"
Sep 09 20:54:15.037992 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.037941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dj9pr" podStartSLOduration=67.291316929 podStartE2EDuration="1m12.037926801s" totalImagesPullingTime="4.746609872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:15.036941708 +0000 UTC m=+71.788486696" watchObservedRunningTime="2026-09-09 20:54:15.037926801 +0000 UTC m=+71.789471707"
Sep 09 20:54:15.292449 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:15.292377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g477p_30921f88-06ce-4cfd-aa6a-07542f3e82a6/node-ca/0.log"
Sep 09 20:54:16.246816 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:16.246764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:16.246816 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:16.246817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:16.247042 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.246891    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.247042 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.246953    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.246938641 +0000 UTC m=+76.998483532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.247042 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.246982    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:16.247042 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.247038    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.247025687 +0000 UTC m=+76.998570578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:16.290957 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:16.290931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmv6d_9e374074-9180-4a2a-91b4-36f06f306cfe/serve-healthcheck-canary/0.log"
Sep 09 20:54:16.650020 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:16.649969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:16.650424 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.650120    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:16.650424 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:16.650189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:24.650174086 +0000 UTC m=+81.401718973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:18.751144 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.751105    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"]
Sep 09 20:54:18.754099 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.754083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"
Sep 09 20:54:18.760057 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.760007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:54:18.760198 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.760167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:18.760273 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.760185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-txhwc\""
Sep 09 20:54:18.761009 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.760994    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-99dnc"]
Sep 09 20:54:18.763569 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.763556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.768802 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.768780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:54:18.769093 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.769079    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:54:18.769146 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.769120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5n5sz\""
Sep 09 20:54:18.773207 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.773190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:18.773296 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.773252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:54:18.778381 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.778362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"]
Sep 09 20:54:18.780142 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.780122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:54:18.788411 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.788390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-99dnc"]
Sep 09 20:54:18.853691 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.853650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"]
Sep 09 20:54:18.856575 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.856559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:18.860739 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.860720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:54:18.860886 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.860758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:54:18.861017 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.860772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qxpdn\""
Sep 09 20:54:18.861183 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.860768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:18.861376 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.861342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"]
Sep 09 20:54:18.864219 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.864202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:18.866291 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72802e85-2836-4fc1-950d-34f43feea95c-serving-cert\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.866371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxpg\" (UniqueName: \"kubernetes.io/projected/bbe35512-fe1b-4791-888a-d29338578816-kube-api-access-lkxpg\") pod \"volume-data-source-validator-695c895bd6-4qs9l\" (UID: \"bbe35512-fe1b-4791-888a-d29338578816\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"
Sep 09 20:54:18.866371 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-config\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.866437 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24lhg\" (UniqueName: \"kubernetes.io/projected/72802e85-2836-4fc1-950d-34f43feea95c-kube-api-access-24lhg\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.866499 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-trusted-ca\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.867005 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kzc97\""
Sep 09 20:54:18.867043 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.867032    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:54:18.867076 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.866990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:54:18.867131 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.867113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:18.869535 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.869516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"]
Sep 09 20:54:18.870564 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.870538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:54:18.876451 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.876431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"]
Sep 09 20:54:18.966888 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24lhg\" (UniqueName: \"kubernetes.io/projected/72802e85-2836-4fc1-950d-34f43feea95c-kube-api-access-24lhg\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxpg\" (UniqueName: \"kubernetes.io/projected/bbe35512-fe1b-4791-888a-d29338578816-kube-api-access-lkxpg\") pod \"volume-data-source-validator-695c895bd6-4qs9l\" (UID: \"bbe35512-fe1b-4791-888a-d29338578816\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxxd\" (UniqueName: \"kubernetes.io/projected/093799ad-20b3-490b-ae75-cae83f0fca45-kube-api-access-6fxxd\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2dw\" (UniqueName: \"kubernetes.io/projected/c61fcf28-b1db-4d45-81cb-0263faef5009-kube-api-access-kb2dw\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-config\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.966994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-trusted-ca\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.967046 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61fcf28-b1db-4d45-81cb-0263faef5009-serving-cert\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:18.967360 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72802e85-2836-4fc1-950d-34f43feea95c-serving-cert\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.967360 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c61fcf28-b1db-4d45-81cb-0263faef5009-config\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:18.967681 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-config\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.967861 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.967842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72802e85-2836-4fc1-950d-34f43feea95c-trusted-ca\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.969452 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.969432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72802e85-2836-4fc1-950d-34f43feea95c-serving-cert\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.980756 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.980735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24lhg\" (UniqueName: \"kubernetes.io/projected/72802e85-2836-4fc1-950d-34f43feea95c-kube-api-access-24lhg\") pod \"console-operator-85b454874b-99dnc\" (UID: \"72802e85-2836-4fc1-950d-34f43feea95c\") " pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:18.980838 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:18.980736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxpg\" (UniqueName: \"kubernetes.io/projected/bbe35512-fe1b-4791-888a-d29338578816-kube-api-access-lkxpg\") pod \"volume-data-source-validator-695c895bd6-4qs9l\" (UID: \"bbe35512-fe1b-4791-888a-d29338578816\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"
Sep 09 20:54:19.062682 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.062608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"
Sep 09 20:54:19.067708 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.067685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxxd\" (UniqueName: \"kubernetes.io/projected/093799ad-20b3-490b-ae75-cae83f0fca45-kube-api-access-6fxxd\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:19.067817 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.067729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2dw\" (UniqueName: \"kubernetes.io/projected/c61fcf28-b1db-4d45-81cb-0263faef5009-kube-api-access-kb2dw\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.067817 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.067754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:19.067817 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.067774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61fcf28-b1db-4d45-81cb-0263faef5009-serving-cert\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.067966 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.067847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c61fcf28-b1db-4d45-81cb-0263faef5009-config\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.067966 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:19.067873    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:54:19.067966 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:19.067938    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls podName:093799ad-20b3-490b-ae75-cae83f0fca45 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:19.567920428 +0000 UTC m=+76.319465330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4nx62" (UID: "093799ad-20b3-490b-ae75-cae83f0fca45") : secret "samples-operator-tls" not found
Sep 09 20:54:19.068374 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.068354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c61fcf28-b1db-4d45-81cb-0263faef5009-config\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.069972 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.069929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c61fcf28-b1db-4d45-81cb-0263faef5009-serving-cert\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.070900 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.070885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:19.080975 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.080955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2dw\" (UniqueName: \"kubernetes.io/projected/c61fcf28-b1db-4d45-81cb-0263faef5009-kube-api-access-kb2dw\") pod \"service-ca-operator-859f94c4b-jjvnl\" (UID: \"c61fcf28-b1db-4d45-81cb-0263faef5009\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.081829 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.081810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxxd\" (UniqueName: \"kubernetes.io/projected/093799ad-20b3-490b-ae75-cae83f0fca45-kube-api-access-6fxxd\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:19.173414 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.173387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"
Sep 09 20:54:19.343869 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.343791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l"]
Sep 09 20:54:19.351204 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.348738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-99dnc"]
Sep 09 20:54:19.351471 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:19.351448    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe35512_fe1b_4791_888a_d29338578816.slice/crio-8919607d7a27c2198a39b20b259a5e95b365a467486ef1fa2f7682d348b9db79 WatchSource:0}: Error finding container 8919607d7a27c2198a39b20b259a5e95b365a467486ef1fa2f7682d348b9db79: Status 404 returned error can't find the container with id 8919607d7a27c2198a39b20b259a5e95b365a467486ef1fa2f7682d348b9db79
Sep 09 20:54:19.442178 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.442136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl"]
Sep 09 20:54:19.446770 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:19.446741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61fcf28_b1db_4d45_81cb_0263faef5009.slice/crio-1f6f71501a9493969e34b6d34668e4e9e0e6af6ad0a3201c91b2f296c95bf4c8 WatchSource:0}: Error finding container 1f6f71501a9493969e34b6d34668e4e9e0e6af6ad0a3201c91b2f296c95bf4c8: Status 404 returned error can't find the container with id 1f6f71501a9493969e34b6d34668e4e9e0e6af6ad0a3201c91b2f296c95bf4c8
Sep 09 20:54:19.571518 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:19.571463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:19.571687 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:19.571614    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:54:19.571687 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:19.571677    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls podName:093799ad-20b3-490b-ae75-cae83f0fca45 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.571662121 +0000 UTC m=+77.323207007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4nx62" (UID: "093799ad-20b3-490b-ae75-cae83f0fca45") : secret "samples-operator-tls" not found
Sep 09 20:54:20.019109 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.019004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l" event={"ID":"bbe35512-fe1b-4791-888a-d29338578816","Type":"ContainerStarted","Data":"8919607d7a27c2198a39b20b259a5e95b365a467486ef1fa2f7682d348b9db79"}
Sep 09 20:54:20.021021 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.020950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl" event={"ID":"c61fcf28-b1db-4d45-81cb-0263faef5009","Type":"ContainerStarted","Data":"1f6f71501a9493969e34b6d34668e4e9e0e6af6ad0a3201c91b2f296c95bf4c8"}
Sep 09 20:54:20.022877 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.022850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-99dnc" event={"ID":"72802e85-2836-4fc1-950d-34f43feea95c","Type":"ContainerStarted","Data":"f9a8271a2deffc891bd64655a6b8d0c5e947b0605d95ad8078939155c0b5845d"}
Sep 09 20:54:20.278397 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.278310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:20.278592 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.278433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:20.278592 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.278455    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:20.278592 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.278552    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.278592 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.278567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.278544416 +0000 UTC m=+85.030089318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:20.278592 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.278593    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.278582719 +0000 UTC m=+85.030127613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.581666 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:20.581582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:20.581830 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.581742    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:54:20.581830 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:20.581814    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls podName:093799ad-20b3-490b-ae75-cae83f0fca45 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:22.581794218 +0000 UTC m=+79.333339110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4nx62" (UID: "093799ad-20b3-490b-ae75-cae83f0fca45") : secret "samples-operator-tls" not found
Sep 09 20:54:21.027846 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:21.027813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l" event={"ID":"bbe35512-fe1b-4791-888a-d29338578816","Type":"ContainerStarted","Data":"2d8ac3c763441a827eb7c47eac4c7709c389e81f63505a465895f57b92033a92"}
Sep 09 20:54:21.050063 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:21.050013    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4qs9l" podStartSLOduration=1.750747617 podStartE2EDuration="3.049991139s" totalImagesPullingTime="1.299243522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:21.048291627 +0000 UTC m=+77.799836535" watchObservedRunningTime="2026-09-09 20:54:21.049991139 +0000 UTC m=+77.801536072"
Sep 09 20:54:22.033806 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.033768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl" event={"ID":"c61fcf28-b1db-4d45-81cb-0263faef5009","Type":"ContainerStarted","Data":"837c30fb3ef8dee9cbefed98cb374e61a09741e982f9eef2dbfb3d2e7f62dc18"}
Sep 09 20:54:22.038564 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.038522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-99dnc" event={"ID":"72802e85-2836-4fc1-950d-34f43feea95c","Type":"ContainerStarted","Data":"ef848fa06b364344d500ed4cf90f91ccf340e55b5c20dfb24230f61fee5cb341"}
Sep 09 20:54:22.038890 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.038850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:22.040175 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.040146    2563 patch_prober.go:28] interesting pod/console-operator-85b454874b-99dnc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.12:8443/readyz\": dial tcp 10.134.0.12:8443: connect: connection refused" start-of-body=
Sep 09 20:54:22.040258 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.040193    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podUID="72802e85-2836-4fc1-950d-34f43feea95c" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.12:8443/readyz\": dial tcp 10.134.0.12:8443: connect: connection refused"
Sep 09 20:54:22.059385 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.059332    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl" podStartSLOduration=1.5877389480000001 podStartE2EDuration="4.059314344s" totalImagesPullingTime="2.471575396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:22.057688775 +0000 UTC m=+78.809233678" watchObservedRunningTime="2026-09-09 20:54:22.059314344 +0000 UTC m=+78.810859255"
Sep 09 20:54:22.099035 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.098981    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podStartSLOduration=1.534509336 podStartE2EDuration="4.098959345s" totalImagesPullingTime="2.564450009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:22.097858129 +0000 UTC m=+78.849403037" watchObservedRunningTime="2026-09-09 20:54:22.098959345 +0000 UTC m=+78.850504256"
Sep 09 20:54:22.596772 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:22.596740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:22.596953 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:22.596856    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:54:22.596953 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:22.596907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls podName:093799ad-20b3-490b-ae75-cae83f0fca45 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:26.596894165 +0000 UTC m=+83.348439052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4nx62" (UID: "093799ad-20b3-490b-ae75-cae83f0fca45") : secret "samples-operator-tls" not found
Sep 09 20:54:23.047956 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:23.047925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/0.log"
Sep 09 20:54:23.048345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:23.047964    2563 generic.go:356] "Generic (PLEG): container finished" podID="72802e85-2836-4fc1-950d-34f43feea95c" containerID="ef848fa06b364344d500ed4cf90f91ccf340e55b5c20dfb24230f61fee5cb341" exitCode=255
Sep 09 20:54:23.048345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:23.047992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-99dnc" event={"ID":"72802e85-2836-4fc1-950d-34f43feea95c","Type":"ContainerDied","Data":"ef848fa06b364344d500ed4cf90f91ccf340e55b5c20dfb24230f61fee5cb341"}
Sep 09 20:54:23.048345 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:23.048223    2563 scope.go:117] "RemoveContainer" containerID="ef848fa06b364344d500ed4cf90f91ccf340e55b5c20dfb24230f61fee5cb341"
Sep 09 20:54:24.051978 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.051948    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 20:54:24.052408 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.052389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/0.log"
Sep 09 20:54:24.052659 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.052434    2563 generic.go:356] "Generic (PLEG): container finished" podID="72802e85-2836-4fc1-950d-34f43feea95c" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987" exitCode=255
Sep 09 20:54:24.052659 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.052464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-99dnc" event={"ID":"72802e85-2836-4fc1-950d-34f43feea95c","Type":"ContainerDied","Data":"fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"}
Sep 09 20:54:24.052659 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.052537    2563 scope.go:117] "RemoveContainer" containerID="ef848fa06b364344d500ed4cf90f91ccf340e55b5c20dfb24230f61fee5cb341"
Sep 09 20:54:24.052789 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.052728    2563 scope.go:117] "RemoveContainer" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"
Sep 09 20:54:24.052922 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:24.052897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-99dnc_openshift-console-operator(72802e85-2836-4fc1-950d-34f43feea95c)\"" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podUID="72802e85-2836-4fc1-950d-34f43feea95c"
Sep 09 20:54:24.712022 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:24.711989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:24.712177 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:24.712132    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:24.712217 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:24.712189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls podName:7615b056-272c-4d62-9ea5-37861f719f42 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:40.712175544 +0000 UTC m=+97.463720435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-lpfrg" (UID: "7615b056-272c-4d62-9ea5-37861f719f42") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:54:25.056940 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:25.056867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 20:54:25.057295 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:25.057168    2563 scope.go:117] "RemoveContainer" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"
Sep 09 20:54:25.057344 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:25.057302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-99dnc_openshift-console-operator(72802e85-2836-4fc1-950d-34f43feea95c)\"" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podUID="72802e85-2836-4fc1-950d-34f43feea95c"
Sep 09 20:54:26.626134 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:26.626010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:26.626513 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:26.626162    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:54:26.626513 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:26.626226    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls podName:093799ad-20b3-490b-ae75-cae83f0fca45 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:34.626209229 +0000 UTC m=+91.377754135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-4nx62" (UID: "093799ad-20b3-490b-ae75-cae83f0fca45") : secret "samples-operator-tls" not found
Sep 09 20:54:28.337259 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:28.337227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:28.337631 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:28.337304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:28.337631 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:28.337389    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.337631 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:28.337462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls podName:d03dc5cf-51f0-4aad-8ed6-00b7916e1f65 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:44.337443908 +0000 UTC m=+101.088988800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2g946" (UID: "d03dc5cf-51f0-4aad-8ed6-00b7916e1f65") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.337898 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:28.337879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:29.072104 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:29.072075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:29.072398 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:29.072385    2563 scope.go:117] "RemoveContainer" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"
Sep 09 20:54:29.072577 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:29.072558    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-99dnc_openshift-console-operator(72802e85-2836-4fc1-950d-34f43feea95c)\"" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podUID="72802e85-2836-4fc1-950d-34f43feea95c"
Sep 09 20:54:32.039282 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:32.039250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:32.039723 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:32.039624    2563 scope.go:117] "RemoveContainer" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"
Sep 09 20:54:32.039777 ip-10-0-138-22 kubenswrapper[2563]: E0909 20:54:32.039759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-99dnc_openshift-console-operator(72802e85-2836-4fc1-950d-34f43feea95c)\"" pod="openshift-console-operator/console-operator-85b454874b-99dnc" podUID="72802e85-2836-4fc1-950d-34f43feea95c"
Sep 09 20:54:34.687004 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:34.686966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:34.689333 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:34.689307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/093799ad-20b3-490b-ae75-cae83f0fca45-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-4nx62\" (UID: \"093799ad-20b3-490b-ae75-cae83f0fca45\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:34.765039 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:34.765007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"
Sep 09 20:54:35.046416 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:35.046335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62"]
Sep 09 20:54:36.089903 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:36.089861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62" event={"ID":"093799ad-20b3-490b-ae75-cae83f0fca45","Type":"ContainerStarted","Data":"e6adc56a08c9c2a7368c2cdac3f80811732d60821a038044cfc404dfb84d339b"}
Sep 09 20:54:37.094809 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:37.094779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62" event={"ID":"093799ad-20b3-490b-ae75-cae83f0fca45","Type":"ContainerStarted","Data":"43fbba483457b63a8504f383142634a45e3cf005b80814ec14ea5d5d2b94494d"}
Sep 09 20:54:38.099553 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:38.099515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62" event={"ID":"093799ad-20b3-490b-ae75-cae83f0fca45","Type":"ContainerStarted","Data":"a7009096ac02d5dac1fefdc5b9f914c3413e4100231cd5f273f57ecd30707ea1"}
Sep 09 20:54:38.121965 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:38.121910    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-4nx62" podStartSLOduration=18.205215964 podStartE2EDuration="20.121893171s" totalImagesPullingTime="1.916677207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:38.120945685 +0000 UTC m=+94.872490591" watchObservedRunningTime="2026-09-09 20:54:38.121893171 +0000 UTC m=+94.873438059"
Sep 09 20:54:40.731921 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:40.731884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:40.734176 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:40.734153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7615b056-272c-4d62-9ea5-37861f719f42-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-lpfrg\" (UID: \"7615b056-272c-4d62-9ea5-37861f719f42\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:40.951147 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:40.951071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"
Sep 09 20:54:41.222615 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.222546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg"]
Sep 09 20:54:41.225432 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:41.225404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7615b056_272c_4d62_9ea5_37861f719f42.slice/crio-82b9d96cdc9699c719e916ac5fe9cbe2c1c6caaf5e5876a4acd387ae756c836a WatchSource:0}: Error finding container 82b9d96cdc9699c719e916ac5fe9cbe2c1c6caaf5e5876a4acd387ae756c836a: Status 404 returned error can't find the container with id 82b9d96cdc9699c719e916ac5fe9cbe2c1c6caaf5e5876a4acd387ae756c836a
Sep 09 20:54:41.537754 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.537677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:41.539926 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.539894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fe439eb-c21c-4593-b642-c613c44889bd-metrics-tls\") pod \"dns-default-xjgzb\" (UID: \"1fe439eb-c21c-4593-b642-c613c44889bd\") " pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:41.554753 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.554727    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:54:41.562495 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.562472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:41.838625 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:41.838555    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xjgzb"]
Sep 09 20:54:41.841214 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:41.841191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe439eb_c21c_4593_b642_c613c44889bd.slice/crio-a0909a6f9bd2a77c28fc7d654c69cc748a09dd4e6d44952c41e0cfd0595ac1a4 WatchSource:0}: Error finding container a0909a6f9bd2a77c28fc7d654c69cc748a09dd4e6d44952c41e0cfd0595ac1a4: Status 404 returned error can't find the container with id a0909a6f9bd2a77c28fc7d654c69cc748a09dd4e6d44952c41e0cfd0595ac1a4
Sep 09 20:54:42.114497 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:42.114396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjgzb" event={"ID":"1fe439eb-c21c-4593-b642-c613c44889bd","Type":"ContainerStarted","Data":"a0909a6f9bd2a77c28fc7d654c69cc748a09dd4e6d44952c41e0cfd0595ac1a4"}
Sep 09 20:54:42.115508 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:42.115466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg" event={"ID":"7615b056-272c-4d62-9ea5-37861f719f42","Type":"ContainerStarted","Data":"82b9d96cdc9699c719e916ac5fe9cbe2c1c6caaf5e5876a4acd387ae756c836a"}
Sep 09 20:54:42.710464 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:42.710435    2563 scope.go:117] "RemoveContainer" containerID="fabaac0a68863225a0a2e4a5c4175e932e9f5980d6d7ba5708d8757425531987"
Sep 09 20:54:43.121696 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:43.121668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 20:54:43.122175 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:43.121770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-99dnc" event={"ID":"72802e85-2836-4fc1-950d-34f43feea95c","Type":"ContainerStarted","Data":"c782e837aa0d3c05fcdc73338a1f69036bb1cae7d6b825fba53f8645e73cd4ef"}
Sep 09 20:54:43.122175 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:43.122141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:43.503000 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:43.502922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-99dnc"
Sep 09 20:54:44.127880 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.127844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjgzb" event={"ID":"1fe439eb-c21c-4593-b642-c613c44889bd","Type":"ContainerStarted","Data":"625e14f0512fb4a41a5e0d4e490525bc76691fcf51faf1e6788da204d35db365"}
Sep 09 20:54:44.128303 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.127904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjgzb" event={"ID":"1fe439eb-c21c-4593-b642-c613c44889bd","Type":"ContainerStarted","Data":"ea2d52516d184fe1f6e74cca153575984175c572ff2411b5a996782fda012946"}
Sep 09 20:54:44.128303 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.127966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:44.152895 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.152850    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xjgzb" podStartSLOduration=65.241546747 podStartE2EDuration="1m7.152836388s" totalImagesPullingTime="1.911289641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:44.152017072 +0000 UTC m=+100.903562002" watchObservedRunningTime="2026-09-09 20:54:44.152836388 +0000 UTC m=+100.904381296"
Sep 09 20:54:44.358479 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.358438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:44.361363 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.361335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d03dc5cf-51f0-4aad-8ed6-00b7916e1f65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2g946\" (UID: \"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65\") " pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:44.384289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:44.384228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2g946"
Sep 09 20:54:45.019095 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:45.019063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2g946"]
Sep 09 20:54:45.021892 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:45.021857    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03dc5cf_51f0_4aad_8ed6_00b7916e1f65.slice/crio-2408b11802c46a3836dc115509474d421c7041ad204166ace293f8edbd2717fa WatchSource:0}: Error finding container 2408b11802c46a3836dc115509474d421c7041ad204166ace293f8edbd2717fa: Status 404 returned error can't find the container with id 2408b11802c46a3836dc115509474d421c7041ad204166ace293f8edbd2717fa
Sep 09 20:54:45.132831 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:45.132800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2g946" event={"ID":"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65","Type":"ContainerStarted","Data":"531807d000689518f8a97ac40cc5de11b1a74b3e2602a6de865e20182d760ffe"}
Sep 09 20:54:45.133209 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:45.132858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2g946" event={"ID":"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65","Type":"ContainerStarted","Data":"2408b11802c46a3836dc115509474d421c7041ad204166ace293f8edbd2717fa"}
Sep 09 20:54:45.134459 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:45.134428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg" event={"ID":"7615b056-272c-4d62-9ea5-37861f719f42","Type":"ContainerStarted","Data":"092028953cb59bd811ecd7e1f988be2ed24315b044a9cd4d57602e02a2cde72d"}
Sep 09 20:54:45.155978 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:45.155929    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-lpfrg" podStartSLOduration=33.580917135 podStartE2EDuration="37.155910079s" totalImagesPullingTime="3.574992944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:45.155054392 +0000 UTC m=+101.906599303" watchObservedRunningTime="2026-09-09 20:54:45.155910079 +0000 UTC m=+101.907454989"
Sep 09 20:54:46.009069 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:46.009035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dj9pr"
Sep 09 20:54:47.143774 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:47.143738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2g946" event={"ID":"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65","Type":"ContainerStarted","Data":"c008052ff91ed446c254c1440bee1ead8628adb34195bcf221a4fe238bb52966"}
Sep 09 20:54:48.770283 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.770173    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"]
Sep 09 20:54:48.792144 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.792121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.804274 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.804247    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:54:48.804393 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.804320    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:54:48.804465 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.804396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:54:48.805558 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.805537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:54:48.805659 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.805642    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:54:48.805833 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.805816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:54:48.805911 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.805852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:54:48.806814 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.806798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"]
Sep 09 20:54:48.893704 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68jf\" (UniqueName: \"kubernetes.io/projected/5e0b2532-3f12-40c9-af15-745d9d52d1e9-kube-api-access-l68jf\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.893873 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.893873 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.893873 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.894014 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.894014 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.893901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0b2532-3f12-40c9-af15-745d9d52d1e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.898966 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.898942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dd7b9cfb-8qvq8"]
Sep 09 20:54:48.916057 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.916037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.920940 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.920916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:54:48.921051 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.920968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:54:48.922691 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.922676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:54:48.923004 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.922989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xb6jr\""
Sep 09 20:54:48.927084 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.927067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:54:48.957761 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.957740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd7b9cfb-8qvq8"]
Sep 09 20:54:48.994853 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-certificates\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.994950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.994950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.994950 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0b2532-3f12-40c9-af15-745d9d52d1e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.995074 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-tls\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995074 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.994994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-trusted-ca\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995074 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7bl\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-kube-api-access-dc7bl\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995074 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68jf\" (UniqueName: \"kubernetes.io/projected/5e0b2532-3f12-40c9-af15-745d9d52d1e9-kube-api-access-l68jf\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.995227 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-image-registry-private-configuration\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995227 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e602467d-8a1d-49b1-bba7-26c731030f1f-ca-trust-extracted\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995306 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-bound-sa-token\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995306 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-installation-pull-secrets\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:48.995306 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.995445 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.995578 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.995557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0b2532-3f12-40c9-af15-745d9d52d1e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.997583 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.997562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.997661 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.997592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.997781 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.997765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-ca\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:48.998021 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:48.997998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0b2532-3f12-40c9-af15-745d9d52d1e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:49.008844 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.008823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68jf\" (UniqueName: \"kubernetes.io/projected/5e0b2532-3f12-40c9-af15-745d9d52d1e9-kube-api-access-l68jf\") pod \"cluster-proxy-proxy-agent-56b77b54b8-7vqq7\" (UID: \"5e0b2532-3f12-40c9-af15-745d9d52d1e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:49.095726 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7bl\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-kube-api-access-dc7bl\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095726 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-image-registry-private-configuration\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095726 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e602467d-8a1d-49b1-bba7-26c731030f1f-ca-trust-extracted\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095943 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-bound-sa-token\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095943 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-installation-pull-secrets\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095943 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-certificates\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095943 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-tls\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.095943 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.095880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-trusted-ca\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.096186 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.096106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e602467d-8a1d-49b1-bba7-26c731030f1f-ca-trust-extracted\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.096678 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.096657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-certificates\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.096785 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.096762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e602467d-8a1d-49b1-bba7-26c731030f1f-trusted-ca\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.098228 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.098202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-registry-tls\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.098322 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.098302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-installation-pull-secrets\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.098372 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.098309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e602467d-8a1d-49b1-bba7-26c731030f1f-image-registry-private-configuration\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.115584 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.115565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-bound-sa-token\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.116417 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.116398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"
Sep 09 20:54:49.117010 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.116992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7bl\" (UniqueName: \"kubernetes.io/projected/e602467d-8a1d-49b1-bba7-26c731030f1f-kube-api-access-dc7bl\") pod \"image-registry-dd7b9cfb-8qvq8\" (UID: \"e602467d-8a1d-49b1-bba7-26c731030f1f\") " pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.157587 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.157557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2g946" event={"ID":"d03dc5cf-51f0-4aad-8ed6-00b7916e1f65","Type":"ContainerStarted","Data":"be7d59c327cbe7b3836cbc332d1c3ccd74840c64e02692e45deb4cef94c0bcd7"}
Sep 09 20:54:49.199510 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.199433    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2g946" podStartSLOduration=34.226181642 podStartE2EDuration="37.199409668s" totalImagesPullingTime="2.973228026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:49.198611085 +0000 UTC m=+105.950155995" watchObservedRunningTime="2026-09-09 20:54:49.199409668 +0000 UTC m=+105.950954578"
Sep 09 20:54:49.224725 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.224698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:49.460355 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.460326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7"]
Sep 09 20:54:49.463727 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:49.463695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e0b2532_3f12_40c9_af15_745d9d52d1e9.slice/crio-6b79857ae30584bb5dc401ab4ed70ce121421794219ab598bbd9c93471f5f50a WatchSource:0}: Error finding container 6b79857ae30584bb5dc401ab4ed70ce121421794219ab598bbd9c93471f5f50a: Status 404 returned error can't find the container with id 6b79857ae30584bb5dc401ab4ed70ce121421794219ab598bbd9c93471f5f50a
Sep 09 20:54:49.526510 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:49.526469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd7b9cfb-8qvq8"]
Sep 09 20:54:49.530131 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:49.530112    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode602467d_8a1d_49b1_bba7_26c731030f1f.slice/crio-31c2e6a2e95e8b5e777ecde97661a6fa162c7d1cdb512a2d01bfef2e3052319d WatchSource:0}: Error finding container 31c2e6a2e95e8b5e777ecde97661a6fa162c7d1cdb512a2d01bfef2e3052319d: Status 404 returned error can't find the container with id 31c2e6a2e95e8b5e777ecde97661a6fa162c7d1cdb512a2d01bfef2e3052319d
Sep 09 20:54:50.161508 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:50.161456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7" event={"ID":"5e0b2532-3f12-40c9-af15-745d9d52d1e9","Type":"ContainerStarted","Data":"6b79857ae30584bb5dc401ab4ed70ce121421794219ab598bbd9c93471f5f50a"}
Sep 09 20:54:50.162939 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:50.162908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8" event={"ID":"e602467d-8a1d-49b1-bba7-26c731030f1f","Type":"ContainerStarted","Data":"9530f1f86a51d920cd32eb64d8a5d76fc1efa72a124c82f89223c318c6faa1c5"}
Sep 09 20:54:50.163095 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:50.162955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8" event={"ID":"e602467d-8a1d-49b1-bba7-26c731030f1f","Type":"ContainerStarted","Data":"31c2e6a2e95e8b5e777ecde97661a6fa162c7d1cdb512a2d01bfef2e3052319d"}
Sep 09 20:54:50.163095 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:50.163010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:54:50.186125 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:50.186071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8" podStartSLOduration=2.186058274 podStartE2EDuration="2.186058274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:50.185022363 +0000 UTC m=+106.936567267" watchObservedRunningTime="2026-09-09 20:54:50.186058274 +0000 UTC m=+106.937603183"
Sep 09 20:54:54.136426 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:54.136386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xjgzb"
Sep 09 20:54:54.179839 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:54.179806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7" event={"ID":"5e0b2532-3f12-40c9-af15-745d9d52d1e9","Type":"ContainerStarted","Data":"eb00a768b311e09814b22112fa03d3a6269a1e4c0c374034eb0a65339792606b"}
Sep 09 20:54:54.180008 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:54.179853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7" event={"ID":"5e0b2532-3f12-40c9-af15-745d9d52d1e9","Type":"ContainerStarted","Data":"fba545b38bf81742fe8caebc725e841e827d1c0b4fd2681b3f8c8257d657cebe"}
Sep 09 20:54:54.180008 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:54.179875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7" event={"ID":"5e0b2532-3f12-40c9-af15-745d9d52d1e9","Type":"ContainerStarted","Data":"fe27db3f5cbef6b866c7a73fa21c4b21079715190c4859136589c060b7f73c5c"}
Sep 09 20:54:54.206504 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:54.206440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56b77b54b8-7vqq7" podStartSLOduration=2.516472487 podStartE2EDuration="6.206423674s" totalImagesPullingTime="3.689951187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:54.20536076 +0000 UTC m=+110.956905670" watchObservedRunningTime="2026-09-09 20:54:54.206423674 +0000 UTC m=+110.957968583"
Sep 09 20:54:57.789928 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.789887    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-48g7d"]
Sep 09 20:54:57.793008 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.792986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.796905 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.796878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:54:57.797831 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.797809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tqnv2\""
Sep 09 20:54:57.797831 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.797825    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:54:57.798008 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.797815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:54:57.798008 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.797845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:54:57.868815 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-tls\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.868957 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-root\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.868957 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-textfile\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.868957 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9w8\" (UniqueName: \"kubernetes.io/projected/f13d6fae-3342-4ee3-ad5a-bd651718ea87-kube-api-access-ms9w8\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.868957 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.869218 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.868992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-sys\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.869218 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.869062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-wtmp\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.869218 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.869099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-accelerators-collector-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.869218 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.869173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-metrics-client-ca\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.969833 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.969796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-sys\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.969988 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.969845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-wtmp\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.969988 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.969907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-sys\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.969988 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.969962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-accelerators-collector-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.969988 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.969983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-wtmp\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-metrics-client-ca\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-tls\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-root\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f13d6fae-3342-4ee3-ad5a-bd651718ea87-root\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-textfile\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9w8\" (UniqueName: \"kubernetes.io/projected/f13d6fae-3342-4ee3-ad5a-bd651718ea87-kube-api-access-ms9w8\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970572 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-textfile\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970657 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-accelerators-collector-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.970715 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.970698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f13d6fae-3342-4ee3-ad5a-bd651718ea87-metrics-client-ca\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.972392 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.972375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-tls\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.972447 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.972396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f13d6fae-3342-4ee3-ad5a-bd651718ea87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:57.979289 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:57.979263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9w8\" (UniqueName: \"kubernetes.io/projected/f13d6fae-3342-4ee3-ad5a-bd651718ea87-kube-api-access-ms9w8\") pod \"node-exporter-48g7d\" (UID: \"f13d6fae-3342-4ee3-ad5a-bd651718ea87\") " pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:58.102536 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:58.102448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48g7d"
Sep 09 20:54:58.111969 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:54:58.111940    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13d6fae_3342_4ee3_ad5a_bd651718ea87.slice/crio-decf08997776d5522dee190e43d63c1a40a5efdaf4e231f3523e1642aa3e3415 WatchSource:0}: Error finding container decf08997776d5522dee190e43d63c1a40a5efdaf4e231f3523e1642aa3e3415: Status 404 returned error can't find the container with id decf08997776d5522dee190e43d63c1a40a5efdaf4e231f3523e1642aa3e3415
Sep 09 20:54:58.195050 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:54:58.195016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48g7d" event={"ID":"f13d6fae-3342-4ee3-ad5a-bd651718ea87","Type":"ContainerStarted","Data":"decf08997776d5522dee190e43d63c1a40a5efdaf4e231f3523e1642aa3e3415"}
Sep 09 20:55:00.204048 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:00.204013    2563 generic.go:356] "Generic (PLEG): container finished" podID="f13d6fae-3342-4ee3-ad5a-bd651718ea87" containerID="066b91521a5e7daa8b9e11ba63b73a67eee3538b915846e86f7676d97ee8a153" exitCode=0
Sep 09 20:55:00.204511 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:00.204074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48g7d" event={"ID":"f13d6fae-3342-4ee3-ad5a-bd651718ea87","Type":"ContainerDied","Data":"066b91521a5e7daa8b9e11ba63b73a67eee3538b915846e86f7676d97ee8a153"}
Sep 09 20:55:01.210170 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:01.210138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48g7d" event={"ID":"f13d6fae-3342-4ee3-ad5a-bd651718ea87","Type":"ContainerStarted","Data":"9ce7061866ac81eaa277b1874c0faae2a6c55bf31404c97760a25269575b6036"}
Sep 09 20:55:01.210601 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:01.210191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48g7d" event={"ID":"f13d6fae-3342-4ee3-ad5a-bd651718ea87","Type":"ContainerStarted","Data":"4feecdecff06368892425661a007ad815f97e209818afcc31aad839f109b42e2"}
Sep 09 20:55:01.240203 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:01.240147    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-48g7d" podStartSLOduration=2.9954524989999998 podStartE2EDuration="4.240129604s" totalImagesPullingTime="1.18353717s" totalInitContainerRuntime="61.139935ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:01.234950128 +0000 UTC m=+117.986495058" watchObservedRunningTime="2026-09-09 20:55:01.240129604 +0000 UTC m=+117.991674514"
Sep 09 20:55:11.171568 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:11.171472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dd7b9cfb-8qvq8"
Sep 09 20:55:13.596330 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:13.596289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:55:13.598562 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:13.598542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf54879-0091-4062-a4b2-22ba7cbfacc3-metrics-certs\") pod \"network-metrics-daemon-rd976\" (UID: \"7bf54879-0091-4062-a4b2-22ba7cbfacc3\") " pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:55:13.833949 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:13.833916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:55:13.841528 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:13.841513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rd976"
Sep 09 20:55:14.114275 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:14.114190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rd976"]
Sep 09 20:55:14.117178 ip-10-0-138-22 kubenswrapper[2563]: W0909 20:55:14.117145    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf54879_0091_4062_a4b2_22ba7cbfacc3.slice/crio-ca230244a7fd0eb9e707a8c1ff37f9d20e76eb361b9c356e96bed55e7ce1c633 WatchSource:0}: Error finding container ca230244a7fd0eb9e707a8c1ff37f9d20e76eb361b9c356e96bed55e7ce1c633: Status 404 returned error can't find the container with id ca230244a7fd0eb9e707a8c1ff37f9d20e76eb361b9c356e96bed55e7ce1c633
Sep 09 20:55:14.252344 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:14.252309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd976" event={"ID":"7bf54879-0091-4062-a4b2-22ba7cbfacc3","Type":"ContainerStarted","Data":"ca230244a7fd0eb9e707a8c1ff37f9d20e76eb361b9c356e96bed55e7ce1c633"}
Sep 09 20:55:17.264148 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:17.264110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd976" event={"ID":"7bf54879-0091-4062-a4b2-22ba7cbfacc3","Type":"ContainerStarted","Data":"bd1cb8e118e9f1294fd45b423676fed034f0b257c5c1080353790491181334f1"}
Sep 09 20:55:17.264596 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:17.264164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rd976" event={"ID":"7bf54879-0091-4062-a4b2-22ba7cbfacc3","Type":"ContainerStarted","Data":"5ad7679e2531ec69a766b41809259c0f91515a0a05e70f53c7421f27095df6b1"}
Sep 09 20:55:17.296088 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:17.296045    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rd976" podStartSLOduration=131.906723522 podStartE2EDuration="2m14.296026501s" totalImagesPullingTime="2.389302979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:17.294598596 +0000 UTC m=+134.046143505" watchObservedRunningTime="2026-09-09 20:55:17.296026501 +0000 UTC m=+134.047571412"
Sep 09 20:55:37.332302 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:37.332213    2563 generic.go:356] "Generic (PLEG): container finished" podID="f146f95f-c995-445f-9d54-dc3931306c22" containerID="4843cc6ff4fa8fcc574ad45eab63679e16310859caf2777dafb352dd463075f6" exitCode=0
Sep 09 20:55:37.332769 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:37.332290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws" event={"ID":"f146f95f-c995-445f-9d54-dc3931306c22","Type":"ContainerDied","Data":"4843cc6ff4fa8fcc574ad45eab63679e16310859caf2777dafb352dd463075f6"}
Sep 09 20:55:37.332769 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:37.332723    2563 scope.go:117] "RemoveContainer" containerID="4843cc6ff4fa8fcc574ad45eab63679e16310859caf2777dafb352dd463075f6"
Sep 09 20:55:38.278653 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:38.278623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjgzb_1fe439eb-c21c-4593-b642-c613c44889bd/dns/0.log"
Sep 09 20:55:38.284463 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:38.284440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjgzb_1fe439eb-c21c-4593-b642-c613c44889bd/kube-rbac-proxy/0.log"
Sep 09 20:55:38.338535 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:38.338500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fckws" event={"ID":"f146f95f-c995-445f-9d54-dc3931306c22","Type":"ContainerStarted","Data":"d940080e565b997d957b3a0105d40c3dc88db306f9216584f0630485253a9675"}
Sep 09 20:55:38.373021 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:38.372991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86nnz_8c8a7501-4a85-41d6-8cb2-b53133ab9993/dns-node-resolver/0.log"
Sep 09 20:55:48.375591 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:48.375556    2563 generic.go:356] "Generic (PLEG): container finished" podID="c61fcf28-b1db-4d45-81cb-0263faef5009" containerID="837c30fb3ef8dee9cbefed98cb374e61a09741e982f9eef2dbfb3d2e7f62dc18" exitCode=0
Sep 09 20:55:48.376006 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:48.375623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl" event={"ID":"c61fcf28-b1db-4d45-81cb-0263faef5009","Type":"ContainerDied","Data":"837c30fb3ef8dee9cbefed98cb374e61a09741e982f9eef2dbfb3d2e7f62dc18"}
Sep 09 20:55:48.376006 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:48.375941    2563 scope.go:117] "RemoveContainer" containerID="837c30fb3ef8dee9cbefed98cb374e61a09741e982f9eef2dbfb3d2e7f62dc18"
Sep 09 20:55:49.381593 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:55:49.381561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jjvnl" event={"ID":"c61fcf28-b1db-4d45-81cb-0263faef5009","Type":"ContainerStarted","Data":"f479087f434bff7e3322bc88f8bde3c99c1d840675bb0c8965e83b5170c408b0"}
Sep 09 20:58:03.619229 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:58:03.619195    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 20:58:03.623995 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:58:03.623947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 20:58:03.629962 ip-10-0-138-22 kubenswrapper[2563]: I0909 20:58:03.629943    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 21:00:14.456640 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.456605    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qq8gr"]
Sep 09 21:00:14.458676 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.458659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.461238 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.461213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 21:00:14.461769 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.461748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 21:00:14.461935 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.461919    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lm9k9\""
Sep 09 21:00:14.469811 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.469788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qq8gr"]
Sep 09 21:00:14.499691 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.499660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.499691 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.499692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqdp\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-kube-api-access-cfqdp\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.600146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.600110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.600146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.600150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqdp\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-kube-api-access-cfqdp\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.609859 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.609828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.610211 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.610189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqdp\" (UniqueName: \"kubernetes.io/projected/55e2fa30-777b-4086-be11-3d2821f4570e-kube-api-access-cfqdp\") pod \"cert-manager-webhook-6486ffd997-qq8gr\" (UID: \"55e2fa30-777b-4086-be11-3d2821f4570e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:14.767668 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:14.767587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:15.060825 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:15.060797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qq8gr"]
Sep 09 21:00:15.063516 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:00:15.063464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55e2fa30_777b_4086_be11_3d2821f4570e.slice/crio-d25cb96ce9ffbc3c747493e061ba47d0817cc251523cb5daa5082ab8ac49337b WatchSource:0}: Error finding container d25cb96ce9ffbc3c747493e061ba47d0817cc251523cb5daa5082ab8ac49337b: Status 404 returned error can't find the container with id d25cb96ce9ffbc3c747493e061ba47d0817cc251523cb5daa5082ab8ac49337b
Sep 09 21:00:15.065470 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:15.065452    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:00:15.291888 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:15.291855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr" event={"ID":"55e2fa30-777b-4086-be11-3d2821f4570e","Type":"ContainerStarted","Data":"d25cb96ce9ffbc3c747493e061ba47d0817cc251523cb5daa5082ab8ac49337b"}
Sep 09 21:00:19.308929 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:19.308832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr" event={"ID":"55e2fa30-777b-4086-be11-3d2821f4570e","Type":"ContainerStarted","Data":"2cbc2fd52060e4f6e417d8617ed0bcd135737cf0950f7722c17d8dab15517513"}
Sep 09 21:00:19.308929 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:19.308884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:19.427934 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:19.427839    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr" podStartSLOduration=1.473345548 podStartE2EDuration="5.427819364s" totalImagesPullingTime="3.954473816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:19.427523634 +0000 UTC m=+436.179068542" watchObservedRunningTime="2026-09-09 21:00:19.427819364 +0000 UTC m=+436.179364272"
Sep 09 21:00:25.314775 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:25.314743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qq8gr"
Sep 09 21:00:26.419474 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.419387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xl424"]
Sep 09 21:00:26.422729 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.422706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.425793 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.425771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2rhwq\""
Sep 09 21:00:26.436874 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.436848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xl424"]
Sep 09 21:00:26.497855 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.497818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.498055 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.497888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqhv\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-kube-api-access-7xqhv\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.598497 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.598451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqhv\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-kube-api-access-7xqhv\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.598650 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.598523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.607264 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.607236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.607371 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.607346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqhv\" (UniqueName: \"kubernetes.io/projected/16880c24-8798-4098-a1ae-d867405aad3e-kube-api-access-7xqhv\") pod \"cert-manager-55b4d954d5-xl424\" (UID: \"16880c24-8798-4098-a1ae-d867405aad3e\") " pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.731904 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.731815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xl424"
Sep 09 21:00:26.999756 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:26.999683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xl424"]
Sep 09 21:00:27.002685 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:00:27.002657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16880c24_8798_4098_a1ae_d867405aad3e.slice/crio-1e93c792055108e7d6fd0186c532e3d8736e611c07d6a541e961ff1cea607263 WatchSource:0}: Error finding container 1e93c792055108e7d6fd0186c532e3d8736e611c07d6a541e961ff1cea607263: Status 404 returned error can't find the container with id 1e93c792055108e7d6fd0186c532e3d8736e611c07d6a541e961ff1cea607263
Sep 09 21:00:27.337632 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:27.337534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xl424" event={"ID":"16880c24-8798-4098-a1ae-d867405aad3e","Type":"ContainerStarted","Data":"497a7a2b45582dc11e56c8767f2c6204dca859c8e7e9d2e87cfdadee2d18e1e0"}
Sep 09 21:00:27.337632 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:27.337591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xl424" event={"ID":"16880c24-8798-4098-a1ae-d867405aad3e","Type":"ContainerStarted","Data":"1e93c792055108e7d6fd0186c532e3d8736e611c07d6a541e961ff1cea607263"}
Sep 09 21:00:27.356061 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:27.356011    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xl424" podStartSLOduration=1.355993338 podStartE2EDuration="1.355993338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:27.355077691 +0000 UTC m=+444.106622600" watchObservedRunningTime="2026-09-09 21:00:27.355993338 +0000 UTC m=+444.107538247"
Sep 09 21:00:46.944261 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.944227    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:00:46.947582 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.947563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.952156 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 21:00:46.952156 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q9gkr\""
Sep 09 21:00:46.952375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952215    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 21:00:46.952375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 21:00:46.952375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:00:46.952375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 21:00:46.953008 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.952989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 21:00:46.959259 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvfq\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.959532 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.959421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:46.973944 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:46.973918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:00:47.060141 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvfq\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.060979 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.060953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.062620 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.062601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.062725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.062653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.062841 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.062824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.062896 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.062873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.085234 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.085204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvfq\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.085758 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.085738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5pd56\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.257040 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.256950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:47.542889 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:47.542812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:00:47.547189 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:00:47.547162    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e59a517_3453_460e_82c3_13b080ca7d59.slice/crio-c0c81c80c3d65cc309af21bf7368388a8ba6485e5ab385779e5e8302c5b73bb4 WatchSource:0}: Error finding container c0c81c80c3d65cc309af21bf7368388a8ba6485e5ab385779e5e8302c5b73bb4: Status 404 returned error can't find the container with id c0c81c80c3d65cc309af21bf7368388a8ba6485e5ab385779e5e8302c5b73bb4
Sep 09 21:00:48.413800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:48.413769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" event={"ID":"6e59a517-3453-460e-82c3-13b080ca7d59","Type":"ContainerStarted","Data":"c0c81c80c3d65cc309af21bf7368388a8ba6485e5ab385779e5e8302c5b73bb4"}
Sep 09 21:00:50.063238 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:50.063195    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:00:50.063630 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:50.063282    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:00:50.423626 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:50.423582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" event={"ID":"6e59a517-3453-460e-82c3-13b080ca7d59","Type":"ContainerStarted","Data":"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"}
Sep 09 21:00:50.423801 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:50.423654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:51.429726 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:51.429697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:00:51.454766 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:00:51.454717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" podStartSLOduration=2.941355308 podStartE2EDuration="5.454700791s" totalImagesPullingTime="2.513345483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:50.453260302 +0000 UTC m=+467.204805211" watchObservedRunningTime="2026-09-09 21:00:51.454700791 +0000 UTC m=+468.206245700"
Sep 09 21:01:16.421381 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.421343    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"]
Sep 09 21:01:16.424468 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.424450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:16.427572 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.427543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:01:16.427694 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.427586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lwn2v\""
Sep 09 21:01:16.427694 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.427592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 21:01:16.427694 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.427631    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:01:16.436516 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.436479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"]
Sep 09 21:01:16.481728 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.481699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszfj\" (UniqueName: \"kubernetes.io/projected/9cc11e15-865e-4993-9986-224da27d9728-kube-api-access-kszfj\") pod \"dns-operator-controller-manager-65dc8f76f5-6vz6s\" (UID: \"9cc11e15-865e-4993-9986-224da27d9728\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:16.582173 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.582132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszfj\" (UniqueName: \"kubernetes.io/projected/9cc11e15-865e-4993-9986-224da27d9728-kube-api-access-kszfj\") pod \"dns-operator-controller-manager-65dc8f76f5-6vz6s\" (UID: \"9cc11e15-865e-4993-9986-224da27d9728\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:16.592769 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.592731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszfj\" (UniqueName: \"kubernetes.io/projected/9cc11e15-865e-4993-9986-224da27d9728-kube-api-access-kszfj\") pod \"dns-operator-controller-manager-65dc8f76f5-6vz6s\" (UID: \"9cc11e15-865e-4993-9986-224da27d9728\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:16.734165 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:16.734087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:17.014557 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:17.014453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"]
Sep 09 21:01:17.017960 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:01:17.017929    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc11e15_865e_4993_9986_224da27d9728.slice/crio-35c7bc4184081ded5872277ca0874daeb67789464e2df18a2349599ccf36a7c6 WatchSource:0}: Error finding container 35c7bc4184081ded5872277ca0874daeb67789464e2df18a2349599ccf36a7c6: Status 404 returned error can't find the container with id 35c7bc4184081ded5872277ca0874daeb67789464e2df18a2349599ccf36a7c6
Sep 09 21:01:17.527552 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:17.527519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s" event={"ID":"9cc11e15-865e-4993-9986-224da27d9728","Type":"ContainerStarted","Data":"35c7bc4184081ded5872277ca0874daeb67789464e2df18a2349599ccf36a7c6"}
Sep 09 21:01:19.438072 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.438039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"]
Sep 09 21:01:19.446988 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.446956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:19.449946 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.449922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t5xhf\""
Sep 09 21:01:19.453609 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.453585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"]
Sep 09 21:01:19.505893 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.505855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdhd\" (UniqueName: \"kubernetes.io/projected/d52ca83e-146d-49b0-a4e8-3df24f7bdc69-kube-api-access-bsdhd\") pod \"authorino-operator-6fc8f864f4-6gzgs\" (UID: \"d52ca83e-146d-49b0-a4e8-3df24f7bdc69\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:19.607227 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.607176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdhd\" (UniqueName: \"kubernetes.io/projected/d52ca83e-146d-49b0-a4e8-3df24f7bdc69-kube-api-access-bsdhd\") pod \"authorino-operator-6fc8f864f4-6gzgs\" (UID: \"d52ca83e-146d-49b0-a4e8-3df24f7bdc69\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:19.618397 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.618365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdhd\" (UniqueName: \"kubernetes.io/projected/d52ca83e-146d-49b0-a4e8-3df24f7bdc69-kube-api-access-bsdhd\") pod \"authorino-operator-6fc8f864f4-6gzgs\" (UID: \"d52ca83e-146d-49b0-a4e8-3df24f7bdc69\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:19.760408 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:19.760324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:20.313563 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:20.313538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"]
Sep 09 21:01:20.315915 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:01:20.315883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ca83e_146d_49b0_a4e8_3df24f7bdc69.slice/crio-98a69b7a7f44f5baefc8b94778ef68c3e4a2700fae1ae9ddcef16fd35474eafd WatchSource:0}: Error finding container 98a69b7a7f44f5baefc8b94778ef68c3e4a2700fae1ae9ddcef16fd35474eafd: Status 404 returned error can't find the container with id 98a69b7a7f44f5baefc8b94778ef68c3e4a2700fae1ae9ddcef16fd35474eafd
Sep 09 21:01:20.541197 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:20.541138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s" event={"ID":"9cc11e15-865e-4993-9986-224da27d9728","Type":"ContainerStarted","Data":"52a5ddc6a429219c8709966102805509b4316f5ad5049f7b08bd4a92d1bdbad8"}
Sep 09 21:01:20.541645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:20.541256    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:20.542463 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:20.542429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs" event={"ID":"d52ca83e-146d-49b0-a4e8-3df24f7bdc69","Type":"ContainerStarted","Data":"98a69b7a7f44f5baefc8b94778ef68c3e4a2700fae1ae9ddcef16fd35474eafd"}
Sep 09 21:01:20.566560 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:20.566448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s" podStartSLOduration=1.476319622 podStartE2EDuration="4.566433421s" totalImagesPullingTime="3.090113799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:20.56551987 +0000 UTC m=+497.317064780" watchObservedRunningTime="2026-09-09 21:01:20.566433421 +0000 UTC m=+497.317978330"
Sep 09 21:01:21.577314 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.577276    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"]
Sep 09 21:01:21.580783 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.580766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:21.586355 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.586332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-snfd5\""
Sep 09 21:01:21.623949 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.623921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2sd\" (UniqueName: \"kubernetes.io/projected/dbeda535-c26d-4595-95b7-7935e4295432-kube-api-access-sj2sd\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wd2k5\" (UID: \"dbeda535-c26d-4595-95b7-7935e4295432\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:21.662120 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.662080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"]
Sep 09 21:01:21.724389 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.724355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2sd\" (UniqueName: \"kubernetes.io/projected/dbeda535-c26d-4595-95b7-7935e4295432-kube-api-access-sj2sd\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wd2k5\" (UID: \"dbeda535-c26d-4595-95b7-7935e4295432\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:21.741423 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.741397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2sd\" (UniqueName: \"kubernetes.io/projected/dbeda535-c26d-4595-95b7-7935e4295432-kube-api-access-sj2sd\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wd2k5\" (UID: \"dbeda535-c26d-4595-95b7-7935e4295432\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:21.891205 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:21.891172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:22.170799 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:22.170775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"]
Sep 09 21:01:22.172320 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:01:22.172293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeda535_c26d_4595_95b7_7935e4295432.slice/crio-5b60809185a3ced3bf32654bf81dbe43d0dde17f4ba489d78a18917b55e8bd00 WatchSource:0}: Error finding container 5b60809185a3ced3bf32654bf81dbe43d0dde17f4ba489d78a18917b55e8bd00: Status 404 returned error can't find the container with id 5b60809185a3ced3bf32654bf81dbe43d0dde17f4ba489d78a18917b55e8bd00
Sep 09 21:01:22.552115 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:22.552033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5" event={"ID":"dbeda535-c26d-4595-95b7-7935e4295432","Type":"ContainerStarted","Data":"5b60809185a3ced3bf32654bf81dbe43d0dde17f4ba489d78a18917b55e8bd00"}
Sep 09 21:01:23.559644 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:23.559605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs" event={"ID":"d52ca83e-146d-49b0-a4e8-3df24f7bdc69","Type":"ContainerStarted","Data":"000ed645ab9628e0c1fb77e7494c25a2b4bbf332e2a2e5a7531e70498518af0a"}
Sep 09 21:01:23.560102 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:23.559711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:23.586999 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:23.586830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs" podStartSLOduration=1.5724253419999998 podStartE2EDuration="4.586806972s" totalImagesPullingTime="3.01438163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:23.585873121 +0000 UTC m=+500.337418030" watchObservedRunningTime="2026-09-09 21:01:23.586806972 +0000 UTC m=+500.338351892"
Sep 09 21:01:24.570869 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:24.570768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5" event={"ID":"dbeda535-c26d-4595-95b7-7935e4295432","Type":"ContainerStarted","Data":"360150bea9ee83b61258ed065006f372f97976002fec69ed15fbe19444e76d2c"}
Sep 09 21:01:24.594365 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:24.594312    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5" podStartSLOduration=1.6154000210000001 podStartE2EDuration="3.594270509s" totalImagesPullingTime="1.978870488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:24.593152027 +0000 UTC m=+501.344696936" watchObservedRunningTime="2026-09-09 21:01:24.594270509 +0000 UTC m=+501.345815417"
Sep 09 21:01:25.575388 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:25.575357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:01:31.548528 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:31.548478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6vz6s"
Sep 09 21:01:34.573471 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:34.573439    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-6gzgs"
Sep 09 21:01:36.582415 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:01:36.582382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wd2k5"
Sep 09 21:02:19.193790 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.193708    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:19.195936 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.195920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.199521 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.199476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bm2b9\""
Sep 09 21:02:19.200071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.200057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 21:02:19.210271 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.210245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:19.283796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.283763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:19.293707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.293674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.293848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.293727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9rf\" (UniqueName: \"kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.395129 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.395092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.395298 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.395136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9rf\" (UniqueName: \"kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.395815 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.395795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.406112 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.406088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9rf\" (UniqueName: \"kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf\") pod \"limitador-limitador-8588749977-hrkbg\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") " pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.505998 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.505906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:19.777586 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.777562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:19.780104 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:02:19.780078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876f4678_4374_4229_bc18_33ed8055872e.slice/crio-1f4037168c6363cc51d648f0b5a4264bb4e2070a16daa34325674ff74ef55f0e WatchSource:0}: Error finding container 1f4037168c6363cc51d648f0b5a4264bb4e2070a16daa34325674ff74ef55f0e: Status 404 returned error can't find the container with id 1f4037168c6363cc51d648f0b5a4264bb4e2070a16daa34325674ff74ef55f0e
Sep 09 21:02:19.802346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:19.802311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" event={"ID":"876f4678-4374-4229-bc18-33ed8055872e","Type":"ContainerStarted","Data":"1f4037168c6363cc51d648f0b5a4264bb4e2070a16daa34325674ff74ef55f0e"}
Sep 09 21:02:21.812111 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:21.812068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" event={"ID":"876f4678-4374-4229-bc18-33ed8055872e","Type":"ContainerStarted","Data":"eee376492974e4d68eb5e829794a6c4c45a5c685e4eeb8af5e39fe4c7c6a593b"}
Sep 09 21:02:21.812577 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:21.812143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:21.834954 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:21.834901    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" podStartSLOduration=1.654319197 podStartE2EDuration="2.83488471s" totalImagesPullingTime="1.180565513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:21.83417462 +0000 UTC m=+558.585719529" watchObservedRunningTime="2026-09-09 21:02:21.83488471 +0000 UTC m=+558.586429652"
Sep 09 21:02:32.334122 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.334081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:32.334746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.334297    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" podUID="876f4678-4374-4229-bc18-33ed8055872e" containerName="limitador" containerID="cri-o://eee376492974e4d68eb5e829794a6c4c45a5c685e4eeb8af5e39fe4c7c6a593b" gracePeriod=30
Sep 09 21:02:32.335161 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.335047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:32.857748 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.857717    2563 generic.go:356] "Generic (PLEG): container finished" podID="876f4678-4374-4229-bc18-33ed8055872e" containerID="eee376492974e4d68eb5e829794a6c4c45a5c685e4eeb8af5e39fe4c7c6a593b" exitCode=0
Sep 09 21:02:32.857901 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.857791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" event={"ID":"876f4678-4374-4229-bc18-33ed8055872e","Type":"ContainerDied","Data":"eee376492974e4d68eb5e829794a6c4c45a5c685e4eeb8af5e39fe4c7c6a593b"}
Sep 09 21:02:32.879408 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:32.879385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:33.012005 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.011914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file\") pod \"876f4678-4374-4229-bc18-33ed8055872e\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") "
Sep 09 21:02:33.012005 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.011979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg9rf\" (UniqueName: \"kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf\") pod \"876f4678-4374-4229-bc18-33ed8055872e\" (UID: \"876f4678-4374-4229-bc18-33ed8055872e\") "
Sep 09 21:02:33.012357 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.012330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file" (OuterVolumeSpecName: "config-file") pod "876f4678-4374-4229-bc18-33ed8055872e" (UID: "876f4678-4374-4229-bc18-33ed8055872e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:33.014375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.014353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf" (OuterVolumeSpecName: "kube-api-access-pg9rf") pod "876f4678-4374-4229-bc18-33ed8055872e" (UID: "876f4678-4374-4229-bc18-33ed8055872e"). InnerVolumeSpecName "kube-api-access-pg9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:33.113185 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.113149    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/876f4678-4374-4229-bc18-33ed8055872e-config-file\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:02:33.113185 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.113180    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9rf\" (UniqueName: \"kubernetes.io/projected/876f4678-4374-4229-bc18-33ed8055872e-kube-api-access-pg9rf\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:02:33.863910 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.863873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg" event={"ID":"876f4678-4374-4229-bc18-33ed8055872e","Type":"ContainerDied","Data":"1f4037168c6363cc51d648f0b5a4264bb4e2070a16daa34325674ff74ef55f0e"}
Sep 09 21:02:33.863910 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.863891    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-hrkbg"
Sep 09 21:02:33.864399 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.863945    2563 scope.go:117] "RemoveContainer" containerID="eee376492974e4d68eb5e829794a6c4c45a5c685e4eeb8af5e39fe4c7c6a593b"
Sep 09 21:02:33.883009 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.882985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:33.891260 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:33.891231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-hrkbg"]
Sep 09 21:02:35.715404 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:02:35.715366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876f4678-4374-4229-bc18-33ed8055872e" path="/var/lib/kubelet/pods/876f4678-4374-4229-bc18-33ed8055872e/volumes"
Sep 09 21:03:03.372956 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.372921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"]
Sep 09 21:03:03.373579 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.373346    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876f4678-4374-4229-bc18-33ed8055872e" containerName="limitador"
Sep 09 21:03:03.373579 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.373366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="876f4678-4374-4229-bc18-33ed8055872e" containerName="limitador"
Sep 09 21:03:03.373579 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.373436    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="876f4678-4374-4229-bc18-33ed8055872e" containerName="limitador"
Sep 09 21:03:03.376315 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.376296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.389293 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.389265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"]
Sep 09 21:03:03.456823 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.456785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.456999 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.456850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.456999 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.456916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.456999 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.456970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.456999 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.456990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.457151 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.457006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.457151 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.457023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zdw\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-kube-api-access-h6zdw\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558334 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558334 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558550 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zdw\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-kube-api-access-h6zdw\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558550 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558550 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.558657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.558594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.559299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.559268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.560871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.560846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.561037 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.561020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.561081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.561032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.561115 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.561081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.566917 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.566895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.567106 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.567088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zdw\" (UniqueName: \"kubernetes.io/projected/0600ee42-4cfa-4b14-8833-1ca6e1e56bc2-kube-api-access-h6zdw\") pod \"istiod-openshift-gateway-5677ccb49b-2gvhz\" (UID: \"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.645296 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.645206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:03:03.652957 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.652929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:03:03.685772 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.685744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:03.963981 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.963909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"]
Sep 09 21:03:03.967515 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:03:03.967457    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0600ee42_4cfa_4b14_8833_1ca6e1e56bc2.slice/crio-2b2991d88004a6540bdc10a8b1e524d379e0aa80644c7a59cbdb0d515f2644e3 WatchSource:0}: Error finding container 2b2991d88004a6540bdc10a8b1e524d379e0aa80644c7a59cbdb0d515f2644e3: Status 404 returned error can't find the container with id 2b2991d88004a6540bdc10a8b1e524d379e0aa80644c7a59cbdb0d515f2644e3
Sep 09 21:03:03.969569 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.969532    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:03:03.969633 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.969602    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:03:03.989820 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:03.989794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz" event={"ID":"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2","Type":"ContainerStarted","Data":"2b2991d88004a6540bdc10a8b1e524d379e0aa80644c7a59cbdb0d515f2644e3"}
Sep 09 21:03:04.996046 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:04.996007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz" event={"ID":"0600ee42-4cfa-4b14-8833-1ca6e1e56bc2","Type":"ContainerStarted","Data":"35219483c779ddc94b89891139aa8aed4af49259a2b993da9043da9072fbf6a8"}
Sep 09 21:03:04.996541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:04.996120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:05.021479 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:05.021428    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz" podStartSLOduration=2.021412378 podStartE2EDuration="2.021412378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:05.019143923 +0000 UTC m=+601.770688848" watchObservedRunningTime="2026-09-09 21:03:05.021412378 +0000 UTC m=+601.772957287"
Sep 09 21:03:06.002644 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.002617    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2gvhz"
Sep 09 21:03:06.066176 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.066148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:03:06.066391 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.066364    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" podUID="6e59a517-3453-460e-82c3-13b080ca7d59" containerName="discovery" containerID="cri-o://5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9" gracePeriod=30
Sep 09 21:03:06.324737 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.324700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:03:06.382692 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhvfq\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.382827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.382827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.382827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.382827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.382827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.383124 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.382853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert\") pod \"6e59a517-3453-460e-82c3-13b080ca7d59\" (UID: \"6e59a517-3453-460e-82c3-13b080ca7d59\") "
Sep 09 21:03:06.383245 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.383195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:06.385742 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:06.385742 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq" (OuterVolumeSpecName: "kube-api-access-jhvfq") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "kube-api-access-jhvfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:06.385927 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:06.385927 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:06.385927 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts" (OuterVolumeSpecName: "cacerts") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:06.385927 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.385828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs" (OuterVolumeSpecName: "local-certs") pod "6e59a517-3453-460e-82c3-13b080ca7d59" (UID: "6e59a517-3453-460e-82c3-13b080ca7d59"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:06.483796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483756    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-ca-configmap\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483787    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-cacerts\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483821    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-kubeconfig\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483834    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-istio-token\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483845    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e59a517-3453-460e-82c3-13b080ca7d59-istio-csr-dns-cert\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483856    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvfq\" (UniqueName: \"kubernetes.io/projected/6e59a517-3453-460e-82c3-13b080ca7d59-kube-api-access-jhvfq\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:06.483961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:06.483868    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59a517-3453-460e-82c3-13b080ca7d59-local-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:03:07.006229 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.006192    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e59a517-3453-460e-82c3-13b080ca7d59" containerID="5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9" exitCode=0
Sep 09 21:03:07.006651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.006235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"
Sep 09 21:03:07.006651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.006283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" event={"ID":"6e59a517-3453-460e-82c3-13b080ca7d59","Type":"ContainerDied","Data":"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"}
Sep 09 21:03:07.006651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.006326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56" event={"ID":"6e59a517-3453-460e-82c3-13b080ca7d59","Type":"ContainerDied","Data":"c0c81c80c3d65cc309af21bf7368388a8ba6485e5ab385779e5e8302c5b73bb4"}
Sep 09 21:03:07.006651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.006345    2563 scope.go:117] "RemoveContainer" containerID="5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"
Sep 09 21:03:07.015082 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.015066    2563 scope.go:117] "RemoveContainer" containerID="5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"
Sep 09 21:03:07.015343 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:03:07.015323    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9\": container with ID starting with 5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9 not found: ID does not exist" containerID="5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"
Sep 09 21:03:07.015396 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.015350    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9"} err="failed to get container status \"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9\": rpc error: code = NotFound desc = could not find container \"5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9\": container with ID starting with 5fd0544ba1d780d2d12b3667dad9ea18d8cbe187ee680dace9cd1565ad1351c9 not found: ID does not exist"
Sep 09 21:03:07.042933 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.042897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:03:07.048888 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.048859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5pd56"]
Sep 09 21:03:07.714965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:07.714929    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e59a517-3453-460e-82c3-13b080ca7d59" path="/var/lib/kubelet/pods/6e59a517-3453-460e-82c3-13b080ca7d59/volumes"
Sep 09 21:03:17.368539 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.368503    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:03:17.368967 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.368797    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e59a517-3453-460e-82c3-13b080ca7d59" containerName="discovery"
Sep 09 21:03:17.368967 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.368809    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e59a517-3453-460e-82c3-13b080ca7d59" containerName="discovery"
Sep 09 21:03:17.368967 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.368860    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e59a517-3453-460e-82c3-13b080ca7d59" containerName="discovery"
Sep 09 21:03:17.373108 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.373090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.376596 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.376573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-252j9\""
Sep 09 21:03:17.376996 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.376965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 21:03:17.377968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.377951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 21:03:17.377968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.377962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 21:03:17.387792 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.387772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:03:17.401297 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.401271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-577669d9c7-2q7rb"]
Sep 09 21:03:17.404558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.404538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.407803 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.407785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hn2v2\""
Sep 09 21:03:17.408042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.408027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 21:03:17.415861 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.415840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-577669d9c7-2q7rb"]
Sep 09 21:03:17.445356 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.445326    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qzpg4"]
Sep 09 21:03:17.448471 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.448455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.451124 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.451101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 21:03:17.451223 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.451101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hcvd6\""
Sep 09 21:03:17.459234 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.459210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qzpg4"]
Sep 09 21:03:17.468215 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.468189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmkn\" (UniqueName: \"kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.468328 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.468242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.468368 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.468324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zk7\" (UniqueName: \"kubernetes.io/projected/07088d44-ad10-4762-87a9-4f93a987986e-kube-api-access-t4zk7\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.468368 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.468364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.569395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmkn\" (UniqueName: \"kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c82d826e-6595-4db5-a8d4-538ed9c5a239-data\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrq88\" (UniqueName: \"kubernetes.io/projected/c82d826e-6595-4db5-a8d4-538ed9c5a239-kube-api-access-hrq88\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zk7\" (UniqueName: \"kubernetes.io/projected/07088d44-ad10-4762-87a9-4f93a987986e-kube-api-access-t4zk7\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:03:17.569567    2563 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 09 21:03:17.569615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.569584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.569838 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:03:17.569629    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert podName:7e3e4567-825f-4324-ab68-2b5a083bdef8 nodeName:}" failed. No retries permitted until 2026-09-09 21:03:18.069609032 +0000 UTC m=+614.821153928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert") pod "kserve-controller-manager-644949856f-hl4mh" (UID: "7e3e4567-825f-4324-ab68-2b5a083bdef8") : secret "kserve-webhook-server-cert" not found
Sep 09 21:03:17.569838 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:03:17.569663    2563 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 21:03:17.569838 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:03:17.569700    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert podName:07088d44-ad10-4762-87a9-4f93a987986e nodeName:}" failed. No retries permitted until 2026-09-09 21:03:18.069687609 +0000 UTC m=+614.821232496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert") pod "llmisvc-controller-manager-577669d9c7-2q7rb" (UID: "07088d44-ad10-4762-87a9-4f93a987986e") : secret "llmisvc-webhook-server-cert" not found
Sep 09 21:03:17.578656 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.578628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zk7\" (UniqueName: \"kubernetes.io/projected/07088d44-ad10-4762-87a9-4f93a987986e-kube-api-access-t4zk7\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:17.578834 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.578809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmkn\" (UniqueName: \"kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:17.670378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.670278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c82d826e-6595-4db5-a8d4-538ed9c5a239-data\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.670378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.670334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq88\" (UniqueName: \"kubernetes.io/projected/c82d826e-6595-4db5-a8d4-538ed9c5a239-kube-api-access-hrq88\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.670701 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.670680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c82d826e-6595-4db5-a8d4-538ed9c5a239-data\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.682991 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.682966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq88\" (UniqueName: \"kubernetes.io/projected/c82d826e-6595-4db5-a8d4-538ed9c5a239-kube-api-access-hrq88\") pod \"seaweedfs-64568bcd49-qzpg4\" (UID: \"c82d826e-6595-4db5-a8d4-538ed9c5a239\") " pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:17.756986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:17.756957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:18.036203 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.033445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qzpg4"]
Sep 09 21:03:18.037725 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:03:18.037697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82d826e_6595_4db5_a8d4_538ed9c5a239.slice/crio-4e0ba3557bb3f1fd520d7712509a35679274a91c988af7d6b39f6b975680eb41 WatchSource:0}: Error finding container 4e0ba3557bb3f1fd520d7712509a35679274a91c988af7d6b39f6b975680eb41: Status 404 returned error can't find the container with id 4e0ba3557bb3f1fd520d7712509a35679274a91c988af7d6b39f6b975680eb41
Sep 09 21:03:18.058109 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.058081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qzpg4" event={"ID":"c82d826e-6595-4db5-a8d4-538ed9c5a239","Type":"ContainerStarted","Data":"4e0ba3557bb3f1fd520d7712509a35679274a91c988af7d6b39f6b975680eb41"}
Sep 09 21:03:18.074028 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.074001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:18.074108 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.074047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:18.076512 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.076465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") pod \"kserve-controller-manager-644949856f-hl4mh\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") " pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:18.076589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.076465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07088d44-ad10-4762-87a9-4f93a987986e-cert\") pod \"llmisvc-controller-manager-577669d9c7-2q7rb\" (UID: \"07088d44-ad10-4762-87a9-4f93a987986e\") " pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:18.283652 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.283618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:18.315543 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.315510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:18.558417 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.558309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:03:18.560731 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:03:18.560704    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3e4567_825f_4324_ab68_2b5a083bdef8.slice/crio-a5b31d53ca4335ed52ea6193bc5050fe82b9ce3a6e822db40188cc67bffa829f WatchSource:0}: Error finding container a5b31d53ca4335ed52ea6193bc5050fe82b9ce3a6e822db40188cc67bffa829f: Status 404 returned error can't find the container with id a5b31d53ca4335ed52ea6193bc5050fe82b9ce3a6e822db40188cc67bffa829f
Sep 09 21:03:18.594334 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:18.594302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-577669d9c7-2q7rb"]
Sep 09 21:03:18.597743 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:03:18.597710    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07088d44_ad10_4762_87a9_4f93a987986e.slice/crio-3115b18e0c1d78d59211e6eba50e6aa5885c6b6ad15fb50c8762670396767888 WatchSource:0}: Error finding container 3115b18e0c1d78d59211e6eba50e6aa5885c6b6ad15fb50c8762670396767888: Status 404 returned error can't find the container with id 3115b18e0c1d78d59211e6eba50e6aa5885c6b6ad15fb50c8762670396767888
Sep 09 21:03:19.065497 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:19.065435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb" event={"ID":"07088d44-ad10-4762-87a9-4f93a987986e","Type":"ContainerStarted","Data":"3115b18e0c1d78d59211e6eba50e6aa5885c6b6ad15fb50c8762670396767888"}
Sep 09 21:03:19.066727 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:19.066700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-hl4mh" event={"ID":"7e3e4567-825f-4324-ab68-2b5a083bdef8","Type":"ContainerStarted","Data":"a5b31d53ca4335ed52ea6193bc5050fe82b9ce3a6e822db40188cc67bffa829f"}
Sep 09 21:03:24.094198 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.094154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb" event={"ID":"07088d44-ad10-4762-87a9-4f93a987986e","Type":"ContainerStarted","Data":"92325096bcd831b3eabdd6c8f9b0dad86991c6b47235586c0184dbc811839d85"}
Sep 09 21:03:24.094683 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.094268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:24.095698 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.095672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qzpg4" event={"ID":"c82d826e-6595-4db5-a8d4-538ed9c5a239","Type":"ContainerStarted","Data":"bac8f86acc89a1081772825ea0e9153832f60b17c9acdc43b29021be791680f9"}
Sep 09 21:03:24.095803 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.095731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:24.097026 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.097004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-hl4mh" event={"ID":"7e3e4567-825f-4324-ab68-2b5a083bdef8","Type":"ContainerStarted","Data":"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"}
Sep 09 21:03:24.097119 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.097083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:03:24.114771 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.114716    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb" podStartSLOduration=2.573516557 podStartE2EDuration="7.114695977s" totalImagesPullingTime="4.54117942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:24.111705647 +0000 UTC m=+620.863250557" watchObservedRunningTime="2026-09-09 21:03:24.114695977 +0000 UTC m=+620.866240889"
Sep 09 21:03:24.129900 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.129845    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-hl4mh" podStartSLOduration=2.590785975 podStartE2EDuration="7.129819155s" totalImagesPullingTime="4.53903318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:24.129785816 +0000 UTC m=+620.881330725" watchObservedRunningTime="2026-09-09 21:03:24.129819155 +0000 UTC m=+620.881364065"
Sep 09 21:03:24.148311 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:24.148256    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qzpg4" podStartSLOduration=1.956620177 podStartE2EDuration="7.148240549s" totalImagesPullingTime="5.191620372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:24.146391241 +0000 UTC m=+620.897936150" watchObservedRunningTime="2026-09-09 21:03:24.148240549 +0000 UTC m=+620.899785458"
Sep 09 21:03:30.103736 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:30.103653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qzpg4"
Sep 09 21:03:55.103599 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:55.103566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-577669d9c7-2q7rb"
Sep 09 21:03:55.106645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:03:55.106624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:04:03.911505 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:03.911461    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:04:03.911925 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:03.911667    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-644949856f-hl4mh" podUID="7e3e4567-825f-4324-ab68-2b5a083bdef8" containerName="manager" containerID="cri-o://1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f" gracePeriod=10
Sep 09 21:04:03.957373 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:03.957340    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-644949856f-6jxms"]
Sep 09 21:04:03.959716 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:03.959695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:03.970091 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:03.970066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-6jxms"]
Sep 09 21:04:04.060690 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.060650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-cert\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.060845 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.060730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvtp\" (UniqueName: \"kubernetes.io/projected/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-kube-api-access-4gvtp\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.155520 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.155481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:04:04.162025 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.162005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-cert\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.162104 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.162043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvtp\" (UniqueName: \"kubernetes.io/projected/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-kube-api-access-4gvtp\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.164290 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.164269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-cert\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.171098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.171075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvtp\" (UniqueName: \"kubernetes.io/projected/7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9-kube-api-access-4gvtp\") pod \"kserve-controller-manager-644949856f-6jxms\" (UID: \"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9\") " pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.263254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.263162    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbmkn\" (UniqueName: \"kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn\") pod \"7e3e4567-825f-4324-ab68-2b5a083bdef8\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") "
Sep 09 21:04:04.263254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.263251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") pod \"7e3e4567-825f-4324-ab68-2b5a083bdef8\" (UID: \"7e3e4567-825f-4324-ab68-2b5a083bdef8\") "
Sep 09 21:04:04.265396 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.265365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert" (OuterVolumeSpecName: "cert") pod "7e3e4567-825f-4324-ab68-2b5a083bdef8" (UID: "7e3e4567-825f-4324-ab68-2b5a083bdef8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:04.265537 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.265509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn" (OuterVolumeSpecName: "kube-api-access-wbmkn") pod "7e3e4567-825f-4324-ab68-2b5a083bdef8" (UID: "7e3e4567-825f-4324-ab68-2b5a083bdef8"). InnerVolumeSpecName "kube-api-access-wbmkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:04.268400 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.268375    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e3e4567-825f-4324-ab68-2b5a083bdef8" containerID="1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f" exitCode=0
Sep 09 21:04:04.268533 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.268422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-hl4mh"
Sep 09 21:04:04.268533 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.268454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-hl4mh" event={"ID":"7e3e4567-825f-4324-ab68-2b5a083bdef8","Type":"ContainerDied","Data":"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"}
Sep 09 21:04:04.268533 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.268527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-hl4mh" event={"ID":"7e3e4567-825f-4324-ab68-2b5a083bdef8","Type":"ContainerDied","Data":"a5b31d53ca4335ed52ea6193bc5050fe82b9ce3a6e822db40188cc67bffa829f"}
Sep 09 21:04:04.268698 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.268552    2563 scope.go:117] "RemoveContainer" containerID="1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"
Sep 09 21:04:04.276572 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.276552    2563 scope.go:117] "RemoveContainer" containerID="1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"
Sep 09 21:04:04.276824 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:04:04.276808    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f\": container with ID starting with 1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f not found: ID does not exist" containerID="1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"
Sep 09 21:04:04.276885 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.276831    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f"} err="failed to get container status \"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f\": rpc error: code = NotFound desc = could not find container \"1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f\": container with ID starting with 1e936d9ae243e1c66affb56759b92d564a5631e1b6261e1e17e0cf2579dffc5f not found: ID does not exist"
Sep 09 21:04:04.293153 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.293126    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:04:04.294836 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.294819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:04.296773 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.296747    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-644949856f-hl4mh"]
Sep 09 21:04:04.363930 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.363904    2563 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e3e4567-825f-4324-ab68-2b5a083bdef8-cert\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.364059 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.363934    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbmkn\" (UniqueName: \"kubernetes.io/projected/7e3e4567-825f-4324-ab68-2b5a083bdef8-kube-api-access-wbmkn\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:04:04.567540 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:04.567432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-644949856f-6jxms"]
Sep 09 21:04:04.570871 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:04:04.570846    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2f40c9_e4f0_4a6e_abdd_50e1acb26af9.slice/crio-0bb8f26a1e381369fb8b496d23f280394696baeca9321195fecdfe64a3022a22 WatchSource:0}: Error finding container 0bb8f26a1e381369fb8b496d23f280394696baeca9321195fecdfe64a3022a22: Status 404 returned error can't find the container with id 0bb8f26a1e381369fb8b496d23f280394696baeca9321195fecdfe64a3022a22
Sep 09 21:04:05.274870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:05.274784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-6jxms" event={"ID":"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9","Type":"ContainerStarted","Data":"71acf19d6ede94b5295a934ac070d750ed3ca65fc333ece04d2bd40df481a349"}
Sep 09 21:04:05.274870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:05.274837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-644949856f-6jxms" event={"ID":"7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9","Type":"ContainerStarted","Data":"0bb8f26a1e381369fb8b496d23f280394696baeca9321195fecdfe64a3022a22"}
Sep 09 21:04:05.275244 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:05.274903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:05.299312 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:05.299266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-644949856f-6jxms" podStartSLOduration=1.9484401980000001 podStartE2EDuration="2.299251318s" totalImagesPullingTime="350.81112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:05.296956302 +0000 UTC m=+662.048501211" watchObservedRunningTime="2026-09-09 21:04:05.299251318 +0000 UTC m=+662.050796226"
Sep 09 21:04:05.715309 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:05.715273    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3e4567-825f-4324-ab68-2b5a083bdef8" path="/var/lib/kubelet/pods/7e3e4567-825f-4324-ab68-2b5a083bdef8/volumes"
Sep 09 21:04:36.282649 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:36.282615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-644949856f-6jxms"
Sep 09 21:04:37.612918 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.612884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-snjp9"]
Sep 09 21:04:37.613277 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.613190    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3e4567-825f-4324-ab68-2b5a083bdef8" containerName="manager"
Sep 09 21:04:37.613277 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.613200    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3e4567-825f-4324-ab68-2b5a083bdef8" containerName="manager"
Sep 09 21:04:37.613277 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.613261    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3e4567-825f-4324-ab68-2b5a083bdef8" containerName="manager"
Sep 09 21:04:37.615349 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.615329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.618367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.618346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 21:04:37.618467 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.618394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-d6cmh\""
Sep 09 21:04:37.636891 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.636867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-snjp9"]
Sep 09 21:04:37.643133 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.643110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-tb6vd"]
Sep 09 21:04:37.645316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.645299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.648134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.648118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 21:04:37.649806 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.649787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 21:04:37.650228 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.650208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-dlm7x\""
Sep 09 21:04:37.674990 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.674964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tb6vd"]
Sep 09 21:04:37.746099 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.746065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985lp\" (UniqueName: \"kubernetes.io/projected/839862e4-4429-4dfe-bac8-1f3c78751267-kube-api-access-985lp\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.746099 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.746101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58af9654-5576-4fd5-b026-280966228ce6-tls-certs\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.746295 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.746129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.746295 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.746151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.746295 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.746202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6cv\" (UniqueName: \"kubernetes.io/projected/58af9654-5576-4fd5-b026-280966228ce6-kube-api-access-4w6cv\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.847443 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.847405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6cv\" (UniqueName: \"kubernetes.io/projected/58af9654-5576-4fd5-b026-280966228ce6-kube-api-access-4w6cv\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.847646 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.847614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985lp\" (UniqueName: \"kubernetes.io/projected/839862e4-4429-4dfe-bac8-1f3c78751267-kube-api-access-985lp\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.847797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.847759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58af9654-5576-4fd5-b026-280966228ce6-tls-certs\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.847857 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.847823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.847899 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.847854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.847957 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:04:37.847930    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 21:04:37.848009 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:04:37.847958    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 21:04:37.848009 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:04:37.847993    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs podName:839862e4-4429-4dfe-bac8-1f3c78751267 nodeName:}" failed. No retries permitted until 2026-09-09 21:04:38.347974103 +0000 UTC m=+695.099519009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs") pod "odh-model-controller-7b966d7984-tb6vd" (UID: "839862e4-4429-4dfe-bac8-1f3c78751267") : secret "odh-model-controller-metrics-tls" not found
Sep 09 21:04:37.848009 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:04:37.848006    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert podName:839862e4-4429-4dfe-bac8-1f3c78751267 nodeName:}" failed. No retries permitted until 2026-09-09 21:04:38.348000374 +0000 UTC m=+695.099545261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert") pod "odh-model-controller-7b966d7984-tb6vd" (UID: "839862e4-4429-4dfe-bac8-1f3c78751267") : secret "odh-model-controller-webhook-cert" not found
Sep 09 21:04:37.850280 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.850259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58af9654-5576-4fd5-b026-280966228ce6-tls-certs\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.856757 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.856732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-985lp\" (UniqueName: \"kubernetes.io/projected/839862e4-4429-4dfe-bac8-1f3c78751267-kube-api-access-985lp\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:37.857208 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.857189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6cv\" (UniqueName: \"kubernetes.io/projected/58af9654-5576-4fd5-b026-280966228ce6-kube-api-access-4w6cv\") pod \"model-serving-api-7b954ffc95-snjp9\" (UID: \"58af9654-5576-4fd5-b026-280966228ce6\") " pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:37.925345 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:37.925314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:38.200717 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.200629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-snjp9"]
Sep 09 21:04:38.204638 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:04:38.204615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58af9654_5576_4fd5_b026_280966228ce6.slice/crio-e2fa68abfae4813d0be892a6dc477f7a9ba2f03b4e9c5170334454bf36a4c717 WatchSource:0}: Error finding container e2fa68abfae4813d0be892a6dc477f7a9ba2f03b4e9c5170334454bf36a4c717: Status 404 returned error can't find the container with id e2fa68abfae4813d0be892a6dc477f7a9ba2f03b4e9c5170334454bf36a4c717
Sep 09 21:04:38.353628 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.353595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:38.353826 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.353636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:38.355984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.355957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-cert\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:38.356098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.355991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839862e4-4429-4dfe-bac8-1f3c78751267-metrics-certs\") pod \"odh-model-controller-7b966d7984-tb6vd\" (UID: \"839862e4-4429-4dfe-bac8-1f3c78751267\") " pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:38.415896 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.415862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-snjp9" event={"ID":"58af9654-5576-4fd5-b026-280966228ce6","Type":"ContainerStarted","Data":"e2fa68abfae4813d0be892a6dc477f7a9ba2f03b4e9c5170334454bf36a4c717"}
Sep 09 21:04:38.555757 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.555671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:38.825362 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:38.825289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tb6vd"]
Sep 09 21:04:38.828315 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:04:38.828288    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839862e4_4429_4dfe_bac8_1f3c78751267.slice/crio-c5f1f78a1e38be62d3c5b5be3aa47baf053fe79b0a245653ffa1be1eb8a518f8 WatchSource:0}: Error finding container c5f1f78a1e38be62d3c5b5be3aa47baf053fe79b0a245653ffa1be1eb8a518f8: Status 404 returned error can't find the container with id c5f1f78a1e38be62d3c5b5be3aa47baf053fe79b0a245653ffa1be1eb8a518f8
Sep 09 21:04:39.423917 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:39.423880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tb6vd" event={"ID":"839862e4-4429-4dfe-bac8-1f3c78751267","Type":"ContainerStarted","Data":"c5f1f78a1e38be62d3c5b5be3aa47baf053fe79b0a245653ffa1be1eb8a518f8"}
Sep 09 21:04:40.431023 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:40.430976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-snjp9" event={"ID":"58af9654-5576-4fd5-b026-280966228ce6","Type":"ContainerStarted","Data":"b65c5170d4fccf8185950f0f15a38303def30a376af1354ac32f36340974b5cb"}
Sep 09 21:04:40.431481 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:40.431084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:40.471663 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:40.471598    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-snjp9" podStartSLOduration=1.660389251 podStartE2EDuration="3.471583126s" totalImagesPullingTime="1.811193875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:40.470892603 +0000 UTC m=+697.222437514" watchObservedRunningTime="2026-09-09 21:04:40.471583126 +0000 UTC m=+697.223128046"
Sep 09 21:04:42.442171 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:42.442130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tb6vd" event={"ID":"839862e4-4429-4dfe-bac8-1f3c78751267","Type":"ContainerStarted","Data":"b1a13fd79fbbe0a5362e2bb0a7cb33129e4a14b4f1ca5683fc1f2ce5a3c23fb0"}
Sep 09 21:04:42.442547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:42.442251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:04:42.459991 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:42.459921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-tb6vd" podStartSLOduration=2.613477246 podStartE2EDuration="5.459905223s" totalImagesPullingTime="2.846427977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:42.459264539 +0000 UTC m=+699.210809451" watchObservedRunningTime="2026-09-09 21:04:42.459905223 +0000 UTC m=+699.211450193"
Sep 09 21:04:51.440120 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:51.440085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-snjp9"
Sep 09 21:04:53.448934 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:04:53.448899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-tb6vd"
Sep 09 21:06:40.611222 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.611150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:06:40.614635 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.614612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.618031 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.618008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-txj6g\""
Sep 09 21:06:40.619560 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.619535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 09 21:06:40.619662 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.619559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 09 21:06:40.619780 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.619686    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 09 21:06:40.619974 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.619956    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-gpd5f\""
Sep 09 21:06:40.634331 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.634308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:06:40.675693 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.675660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4pm\" (UniqueName: \"kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.675837 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.675707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.777063 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.777028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4pm\" (UniqueName: \"kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.777239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.777082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.779586 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.779561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.789619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.789596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4pm\" (UniqueName: \"kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:40.925625 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:40.925590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:06:41.198846 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:41.198820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:06:41.201226 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:06:41.201200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eb5cba_78ec_4b25_b383_0091cb851a02.slice/crio-7debfe572cd72caaf70516621fb072cbf4d65b4dfed4b3af98a6cffdba36046f WatchSource:0}: Error finding container 7debfe572cd72caaf70516621fb072cbf4d65b4dfed4b3af98a6cffdba36046f: Status 404 returned error can't find the container with id 7debfe572cd72caaf70516621fb072cbf4d65b4dfed4b3af98a6cffdba36046f
Sep 09 21:06:41.203127 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:41.203110    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:06:41.977180 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:41.977102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" event={"ID":"26eb5cba-78ec-4b25-b383-0091cb851a02","Type":"ContainerStarted","Data":"7debfe572cd72caaf70516621fb072cbf4d65b4dfed4b3af98a6cffdba36046f"}
Sep 09 21:06:43.988447 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:43.988347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" event={"ID":"26eb5cba-78ec-4b25-b383-0091cb851a02","Type":"ContainerStarted","Data":"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"}
Sep 09 21:06:43.988897 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:06:43.988443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:07:14.994608 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:14.994575    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:07:15.019860 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:15.019800    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podStartSLOduration=32.524532901 podStartE2EDuration="35.01978244s" totalImagesPullingTime="2.495249539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:44.014585783 +0000 UTC m=+820.766130717" watchObservedRunningTime="2026-09-09 21:07:15.01978244 +0000 UTC m=+851.771327350"
Sep 09 21:07:21.670399 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:21.670369    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:07:24.994110 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:24.994073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:34.994411 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:34.994365    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:36.690141 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:36.690100    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" containerID="cri-o://53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1" gracePeriod=45
Sep 09 21:07:44.994704 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:44.994666    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:44.995106 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:44.994784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:07:54.994304 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:07:54.994264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:03.683959 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:03.683885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:08:03.684415 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:03.683979    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:08:04.994219 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:04.994181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:06.938965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:06.938930    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:08:06.991779 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:06.991705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr4pm\" (UniqueName: \"kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm\") pod \"26eb5cba-78ec-4b25-b383-0091cb851a02\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") "
Sep 09 21:08:06.991779 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:06.991761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs\") pod \"26eb5cba-78ec-4b25-b383-0091cb851a02\" (UID: \"26eb5cba-78ec-4b25-b383-0091cb851a02\") "
Sep 09 21:08:06.993982 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:06.993951    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm" (OuterVolumeSpecName: "kube-api-access-qr4pm") pod "26eb5cba-78ec-4b25-b383-0091cb851a02" (UID: "26eb5cba-78ec-4b25-b383-0091cb851a02"). InnerVolumeSpecName "kube-api-access-qr4pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:06.993982 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:06.993960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26eb5cba-78ec-4b25-b383-0091cb851a02" (UID: "26eb5cba-78ec-4b25-b383-0091cb851a02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:07.092978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.092922    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4pm\" (UniqueName: \"kubernetes.io/projected/26eb5cba-78ec-4b25-b383-0091cb851a02-kube-api-access-qr4pm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:08:07.092978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.092971    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26eb5cba-78ec-4b25-b383-0091cb851a02-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:08:07.362524 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.362404    2563 generic.go:356] "Generic (PLEG): container finished" podID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerID="53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1" exitCode=0
Sep 09 21:08:07.362524 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.362450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" event={"ID":"26eb5cba-78ec-4b25-b383-0091cb851a02","Type":"ContainerDied","Data":"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"}
Sep 09 21:08:07.362524 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.362456    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"
Sep 09 21:08:07.362800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.362543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x" event={"ID":"26eb5cba-78ec-4b25-b383-0091cb851a02","Type":"ContainerDied","Data":"7debfe572cd72caaf70516621fb072cbf4d65b4dfed4b3af98a6cffdba36046f"}
Sep 09 21:08:07.362800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.362565    2563 scope.go:117] "RemoveContainer" containerID="53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"
Sep 09 21:08:07.370959 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.370941    2563 scope.go:117] "RemoveContainer" containerID="53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"
Sep 09 21:08:07.371229 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:08:07.371208    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1\": container with ID starting with 53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1 not found: ID does not exist" containerID="53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"
Sep 09 21:08:07.371275 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.371238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1"} err="failed to get container status \"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1\": rpc error: code = NotFound desc = could not find container \"53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1\": container with ID starting with 53e327de51cf76afabf030db17dd1b2ffcdcf7d67ac6ebbe2286b387ce75a4e1 not found: ID does not exist"
Sep 09 21:08:07.404853 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.404822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:08:07.408374 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.408352    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h265x"]
Sep 09 21:08:07.715526 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:07.715471    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" path="/var/lib/kubelet/pods/26eb5cba-78ec-4b25-b383-0091cb851a02/volumes"
Sep 09 21:08:19.697787 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.697750    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:08:19.698192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.698085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main"
Sep 09 21:08:19.698192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.698099    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main"
Sep 09 21:08:19.698192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.698164    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26eb5cba-78ec-4b25-b383-0091cb851a02" containerName="main"
Sep 09 21:08:19.700979 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.700959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.705965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.705937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-7l8dn\""
Sep 09 21:08:19.706098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.705963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 09 21:08:19.706098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.705988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-mcncw\""
Sep 09 21:08:19.706098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.706008    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 09 21:08:19.706262 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.706095    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 09 21:08:19.722156 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.722126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:08:19.800623 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.800586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.800823 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.800629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnps8\" (UniqueName: \"kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.901647 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.901612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.901818 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.901657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnps8\" (UniqueName: \"kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.904181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.904152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:19.912667 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:19.912643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnps8\" (UniqueName: \"kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:20.011571 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.011469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:20.284960 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.284889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:08:20.288351 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:08:20.288326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6c1610_130d_4e18_bd62_73ce65c203ed.slice/crio-ef6dda00e2551f1ff9b3b712f127d7c13e269203fa06a7f5bcea5cedbb703116 WatchSource:0}: Error finding container ef6dda00e2551f1ff9b3b712f127d7c13e269203fa06a7f5bcea5cedbb703116: Status 404 returned error can't find the container with id ef6dda00e2551f1ff9b3b712f127d7c13e269203fa06a7f5bcea5cedbb703116
Sep 09 21:08:20.431274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.431235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" event={"ID":"7e6c1610-130d-4e18-bd62-73ce65c203ed","Type":"ContainerStarted","Data":"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"}
Sep 09 21:08:20.431457 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.431298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" event={"ID":"7e6c1610-130d-4e18-bd62-73ce65c203ed","Type":"ContainerStarted","Data":"ef6dda00e2551f1ff9b3b712f127d7c13e269203fa06a7f5bcea5cedbb703116"}
Sep 09 21:08:20.431457 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.431359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:20.456621 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:20.456573    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podStartSLOduration=1.45655675 podStartE2EDuration="1.45655675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:20.45478858 +0000 UTC m=+917.206333489" watchObservedRunningTime="2026-09-09 21:08:20.45655675 +0000 UTC m=+917.208101659"
Sep 09 21:08:51.437299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:51.437269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:08:57.740224 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:08:57.740191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:09:01.437319 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:01.437280    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:11.437260 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:11.437218    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:12.759767 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:12.759719    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" containerID="cri-o://b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45" gracePeriod=45
Sep 09 21:09:21.437245 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:21.437205    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:21.437713 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:21.437311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:09:31.437378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:31.437289    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:32.302079 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.302033    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:09:32.305571 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.305476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.310814 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.310786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 21:09:32.312428 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.312402    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 21:09:32.312584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.312436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 21:09:32.312584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.312480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vvgkn\""
Sep 09 21:09:32.312584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.312398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-mf5dz\""
Sep 09 21:09:32.328474 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.328446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:09:32.401353 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.401316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.401353 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.401359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqf5\" (UniqueName: \"kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.502501 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.502457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.502888 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.502525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqf5\" (UniqueName: \"kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.505081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.505051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.511851 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.511830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqf5\" (UniqueName: \"kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.615818 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.615788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:32.896511 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:32.896461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:09:32.899290 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:09:32.899262    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cae042_2aec_4dc7_95e8_7aea09ceafd9.slice/crio-5b4c623b756b56c52d2a8e44c96dc51f423923ebb74694115dd34bde4399c0b3 WatchSource:0}: Error finding container 5b4c623b756b56c52d2a8e44c96dc51f423923ebb74694115dd34bde4399c0b3: Status 404 returned error can't find the container with id 5b4c623b756b56c52d2a8e44c96dc51f423923ebb74694115dd34bde4399c0b3
Sep 09 21:09:33.774940 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:33.774907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" event={"ID":"e7cae042-2aec-4dc7-95e8-7aea09ceafd9","Type":"ContainerStarted","Data":"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"}
Sep 09 21:09:33.775391 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:33.774968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:09:33.775391 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:33.774994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" event={"ID":"e7cae042-2aec-4dc7-95e8-7aea09ceafd9","Type":"ContainerStarted","Data":"5b4c623b756b56c52d2a8e44c96dc51f423923ebb74694115dd34bde4399c0b3"}
Sep 09 21:09:33.807732 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:33.807686    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podStartSLOduration=1.807671076 podStartE2EDuration="1.807671076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:33.805883068 +0000 UTC m=+990.557427999" watchObservedRunningTime="2026-09-09 21:09:33.807671076 +0000 UTC m=+990.559215984"
Sep 09 21:09:41.437361 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:41.437313    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:43.015223 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.015193    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:09:43.074434 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.074357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs\") pod \"7e6c1610-130d-4e18-bd62-73ce65c203ed\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") "
Sep 09 21:09:43.076477 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.076456    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e6c1610-130d-4e18-bd62-73ce65c203ed" (UID: "7e6c1610-130d-4e18-bd62-73ce65c203ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:43.181381 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.176006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnps8\" (UniqueName: \"kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8\") pod \"7e6c1610-130d-4e18-bd62-73ce65c203ed\" (UID: \"7e6c1610-130d-4e18-bd62-73ce65c203ed\") "
Sep 09 21:09:43.181381 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.176474    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6c1610-130d-4e18-bd62-73ce65c203ed-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:09:43.183097 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.183069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8" (OuterVolumeSpecName: "kube-api-access-wnps8") pod "7e6c1610-130d-4e18-bd62-73ce65c203ed" (UID: "7e6c1610-130d-4e18-bd62-73ce65c203ed"). InnerVolumeSpecName "kube-api-access-wnps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:43.277210 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.277169    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnps8\" (UniqueName: \"kubernetes.io/projected/7e6c1610-130d-4e18-bd62-73ce65c203ed-kube-api-access-wnps8\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:09:43.825084 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.825052    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerID="b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45" exitCode=0
Sep 09 21:09:43.825254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.825109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"
Sep 09 21:09:43.825254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.825143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" event={"ID":"7e6c1610-130d-4e18-bd62-73ce65c203ed","Type":"ContainerDied","Data":"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"}
Sep 09 21:09:43.825254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.825198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9" event={"ID":"7e6c1610-130d-4e18-bd62-73ce65c203ed","Type":"ContainerDied","Data":"ef6dda00e2551f1ff9b3b712f127d7c13e269203fa06a7f5bcea5cedbb703116"}
Sep 09 21:09:43.825254 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.825221    2563 scope.go:117] "RemoveContainer" containerID="b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"
Sep 09 21:09:43.833905 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.833881    2563 scope.go:117] "RemoveContainer" containerID="b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"
Sep 09 21:09:43.834176 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:09:43.834156    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45\": container with ID starting with b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45 not found: ID does not exist" containerID="b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"
Sep 09 21:09:43.834251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.834187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45"} err="failed to get container status \"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45\": rpc error: code = NotFound desc = could not find container \"b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45\": container with ID starting with b18e092366ee280dcb7d2152cf578f6a2f7dd7d12fde0cb74ea3e02ff8a0fc45 not found: ID does not exist"
Sep 09 21:09:43.844311 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.844289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:09:43.847629 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:43.847607    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lq4n9"]
Sep 09 21:09:45.715883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:09:45.715842    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" path="/var/lib/kubelet/pods/7e6c1610-130d-4e18-bd62-73ce65c203ed/volumes"
Sep 09 21:10:04.781369 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:10:04.781340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:11:07.581141 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.581108    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:11:07.581610 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.581538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main"
Sep 09 21:11:07.581610 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.581554    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main"
Sep 09 21:11:07.581610 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.581611    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6c1610-130d-4e18-bd62-73ce65c203ed" containerName="main"
Sep 09 21:11:07.584304 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.584282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.588777 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.588747    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 09 21:11:07.588898 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.588829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 09 21:11:07.588898 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.588867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-thq9k\""
Sep 09 21:11:07.588898 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.588890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-wfxcq\""
Sep 09 21:11:07.589040 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.589018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 09 21:11:07.594251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.594230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:11:07.674260 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.674228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.674429 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.674268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqj7r\" (UniqueName: \"kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.775323 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.775288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.775529 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.775330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqj7r\" (UniqueName: \"kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.777711 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.777691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.786146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.786125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqj7r\" (UniqueName: \"kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:07.894872 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:07.894849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:08.170804 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:08.170781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:11:08.173136 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:11:08.173109    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11652f9_f1d0_40c2_b381_9f2890ee9461.slice/crio-d66d92d817bc4b10b1020c4239291cde446d7f6e0a8798a225d653527f6ed1e0 WatchSource:0}: Error finding container d66d92d817bc4b10b1020c4239291cde446d7f6e0a8798a225d653527f6ed1e0: Status 404 returned error can't find the container with id d66d92d817bc4b10b1020c4239291cde446d7f6e0a8798a225d653527f6ed1e0
Sep 09 21:11:08.194295 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:08.194266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" event={"ID":"e11652f9-f1d0-40c2-b381-9f2890ee9461","Type":"ContainerStarted","Data":"d66d92d817bc4b10b1020c4239291cde446d7f6e0a8798a225d653527f6ed1e0"}
Sep 09 21:11:09.199874 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:09.199840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" event={"ID":"e11652f9-f1d0-40c2-b381-9f2890ee9461","Type":"ContainerStarted","Data":"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"}
Sep 09 21:11:09.200258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:09.199909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:09.219718 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:09.219660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podStartSLOduration=2.21964591 podStartE2EDuration="2.21964591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:09.218688621 +0000 UTC m=+1085.970233546" watchObservedRunningTime="2026-09-09 21:11:09.21964591 +0000 UTC m=+1085.971190846"
Sep 09 21:11:22.118649 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:22.118603    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:11:24.781150 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:24.781113    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:34.781307 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:34.781261    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:37.138134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:37.138088    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" containerID="cri-o://66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7" gracePeriod=45
Sep 09 21:11:40.205868 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:40.205837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:11:44.781518 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:44.781462    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:44.781886 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:44.781588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:11:54.781121 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:11:54.781086    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:04.781163 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:04.781122    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:07.379830 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.379803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:12:07.456071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.455990    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerID="66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7" exitCode=0
Sep 09 21:12:07.456071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.456032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"
Sep 09 21:12:07.456071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.456041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" event={"ID":"e7cae042-2aec-4dc7-95e8-7aea09ceafd9","Type":"ContainerDied","Data":"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"}
Sep 09 21:12:07.456373 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.456127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b" event={"ID":"e7cae042-2aec-4dc7-95e8-7aea09ceafd9","Type":"ContainerDied","Data":"5b4c623b756b56c52d2a8e44c96dc51f423923ebb74694115dd34bde4399c0b3"}
Sep 09 21:12:07.456373 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.456160    2563 scope.go:117] "RemoveContainer" containerID="66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"
Sep 09 21:12:07.463991 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.463961    2563 scope.go:117] "RemoveContainer" containerID="66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"
Sep 09 21:12:07.464305 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:12:07.464287    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7\": container with ID starting with 66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7 not found: ID does not exist" containerID="66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"
Sep 09 21:12:07.464366 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.464315    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7"} err="failed to get container status \"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7\": rpc error: code = NotFound desc = could not find container \"66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7\": container with ID starting with 66795244d75f14c22c70f9661b6e8aece512705d26697baa84615cd40fa284d7 not found: ID does not exist"
Sep 09 21:12:07.481664 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.481644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhqf5\" (UniqueName: \"kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5\") pod \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") "
Sep 09 21:12:07.481761 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.481671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs\") pod \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\" (UID: \"e7cae042-2aec-4dc7-95e8-7aea09ceafd9\") "
Sep 09 21:12:07.483801 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.483775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5" (OuterVolumeSpecName: "kube-api-access-nhqf5") pod "e7cae042-2aec-4dc7-95e8-7aea09ceafd9" (UID: "e7cae042-2aec-4dc7-95e8-7aea09ceafd9"). InnerVolumeSpecName "kube-api-access-nhqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:07.483883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.483804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7cae042-2aec-4dc7-95e8-7aea09ceafd9" (UID: "e7cae042-2aec-4dc7-95e8-7aea09ceafd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:07.583031 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.583002    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqf5\" (UniqueName: \"kubernetes.io/projected/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-kube-api-access-nhqf5\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:12:07.583031 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.583027    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cae042-2aec-4dc7-95e8-7aea09ceafd9-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:12:07.779863 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.779836    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:12:07.783653 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:07.783632    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherwd5b"]
Sep 09 21:12:09.715141 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:09.715106    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" path="/var/lib/kubelet/pods/e7cae042-2aec-4dc7-95e8-7aea09ceafd9/volumes"
Sep 09 21:12:19.401275 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.401238    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:12:19.401651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.401584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main"
Sep 09 21:12:19.401651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.401596    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main"
Sep 09 21:12:19.401651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.401645    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7cae042-2aec-4dc7-95e8-7aea09ceafd9" containerName="main"
Sep 09 21:12:19.405888 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.405867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.411764 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.411739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-d7csd\""
Sep 09 21:12:19.411885 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.411778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-nrxw9\""
Sep 09 21:12:19.411885 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.411845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 09 21:12:19.412008 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.411900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 09 21:12:19.412008 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.411913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 09 21:12:19.422247 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.422224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:12:19.580857 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.580825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.581032 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.580890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9w5b\" (UniqueName: \"kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.682365 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.682281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9w5b\" (UniqueName: \"kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.682365 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.682340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.684933 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.684906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.704821 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.704790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9w5b\" (UniqueName: \"kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.716024 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.715996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:19.991875 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:19.991847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:12:19.993383 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:12:19.993357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b9ee4c_0e90_485c_8419_873d083591bb.slice/crio-4d8bf1626474c3177995c81eb4e16aaa02619d764bc699d3b909f0670ff900d5 WatchSource:0}: Error finding container 4d8bf1626474c3177995c81eb4e16aaa02619d764bc699d3b909f0670ff900d5: Status 404 returned error can't find the container with id 4d8bf1626474c3177995c81eb4e16aaa02619d764bc699d3b909f0670ff900d5
Sep 09 21:12:20.512827 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:20.512795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" event={"ID":"23b9ee4c-0e90-485c-8419-873d083591bb","Type":"ContainerStarted","Data":"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"}
Sep 09 21:12:20.513250 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:20.512848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" event={"ID":"23b9ee4c-0e90-485c-8419-873d083591bb","Type":"ContainerStarted","Data":"4d8bf1626474c3177995c81eb4e16aaa02619d764bc699d3b909f0670ff900d5"}
Sep 09 21:12:20.513250 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:20.512903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:20.537813 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:20.537762    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podStartSLOduration=1.537747727 podStartE2EDuration="1.537747727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:20.536265097 +0000 UTC m=+1157.287810005" watchObservedRunningTime="2026-09-09 21:12:20.537747727 +0000 UTC m=+1157.289292636"
Sep 09 21:12:51.519133 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:51.519101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:12:53.200966 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:12:53.200935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:13:01.518861 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:01.518816    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:03.710286 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:03.710253    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:13:03.712190 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:03.712162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:13:08.223554 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:08.223477    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main" containerID="cri-o://ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09" gracePeriod=45
Sep 09 21:13:11.519479 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:11.519441    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:12.906363 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:12.906331    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:13:20.205479 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:20.205434    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:21.518817 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:21.518772    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:21.519270 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:21.518873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:13:27.925949 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:27.925892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main" containerID="cri-o://1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f" gracePeriod=45
Sep 09 21:13:30.206072 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:30.206033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:31.519250 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:31.519210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:38.464331 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.464306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:13:38.516403 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.516324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9w5b\" (UniqueName: \"kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b\") pod \"23b9ee4c-0e90-485c-8419-873d083591bb\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") "
Sep 09 21:13:38.516403 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.516370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs\") pod \"23b9ee4c-0e90-485c-8419-873d083591bb\" (UID: \"23b9ee4c-0e90-485c-8419-873d083591bb\") "
Sep 09 21:13:38.518590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.518559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b" (OuterVolumeSpecName: "kube-api-access-m9w5b") pod "23b9ee4c-0e90-485c-8419-873d083591bb" (UID: "23b9ee4c-0e90-485c-8419-873d083591bb"). InnerVolumeSpecName "kube-api-access-m9w5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:38.518590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.518575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23b9ee4c-0e90-485c-8419-873d083591bb" (UID: "23b9ee4c-0e90-485c-8419-873d083591bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:38.617576 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.617543    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9ee4c-0e90-485c-8419-873d083591bb-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:13:38.617576 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.617574    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9w5b\" (UniqueName: \"kubernetes.io/projected/23b9ee4c-0e90-485c-8419-873d083591bb-kube-api-access-m9w5b\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:13:38.863839 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.863759    2563 generic.go:356] "Generic (PLEG): container finished" podID="23b9ee4c-0e90-485c-8419-873d083591bb" containerID="ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09" exitCode=0
Sep 09 21:13:38.863839 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.863806    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"
Sep 09 21:13:38.863998 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.863854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" event={"ID":"23b9ee4c-0e90-485c-8419-873d083591bb","Type":"ContainerDied","Data":"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"}
Sep 09 21:13:38.863998 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.863905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj" event={"ID":"23b9ee4c-0e90-485c-8419-873d083591bb","Type":"ContainerDied","Data":"4d8bf1626474c3177995c81eb4e16aaa02619d764bc699d3b909f0670ff900d5"}
Sep 09 21:13:38.863998 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.863928    2563 scope.go:117] "RemoveContainer" containerID="ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"
Sep 09 21:13:38.872263 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.872247    2563 scope.go:117] "RemoveContainer" containerID="ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"
Sep 09 21:13:38.872555 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:13:38.872535    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09\": container with ID starting with ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09 not found: ID does not exist" containerID="ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"
Sep 09 21:13:38.872613 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.872563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09"} err="failed to get container status \"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09\": rpc error: code = NotFound desc = could not find container \"ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09\": container with ID starting with ff1885eb59b3a00219bebd5bad922daf2ce5daf27d3d84e21c73621a75cf3c09 not found: ID does not exist"
Sep 09 21:13:38.888365 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.888334    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:13:38.895416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:38.895396    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbnfnlj"]
Sep 09 21:13:39.716299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:39.716263    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" path="/var/lib/kubelet/pods/23b9ee4c-0e90-485c-8419-873d083591bb/volumes"
Sep 09 21:13:40.206326 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:40.206288    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:40.206482 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:40.206402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:13:50.205940 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:50.205896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:13:51.976943 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.976903    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:13:51.979107 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.977267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main"
Sep 09 21:13:51.979107 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.977279    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main"
Sep 09 21:13:51.979107 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.977345    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b9ee4c-0e90-485c-8419-873d083591bb" containerName="main"
Sep 09 21:13:51.980048 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.980028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:51.984069 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.984047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-vsjdz\""
Sep 09 21:13:51.984296 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.984282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-5qznp\""
Sep 09 21:13:51.984405 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.984385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 09 21:13:51.984453 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.984441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 09 21:13:51.984671 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.984654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 09 21:13:51.989891 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:51.989867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:13:52.028973 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.028940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.029129 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.028988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhsfs\" (UniqueName: \"kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.130265 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.130226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.130435 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.130276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsfs\" (UniqueName: \"kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.132770 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.132750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.140187 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.140116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhsfs\" (UniqueName: \"kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.291096 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.291022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.600578 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.600550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:13:52.602459 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:13:52.602431    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444081c8_bcb9_49d4_8823_d4e8c0810779.slice/crio-8762bcb48db1d504456603795d1a95d28a7ac124c59b7a3dbf6680c9a90f1612 WatchSource:0}: Error finding container 8762bcb48db1d504456603795d1a95d28a7ac124c59b7a3dbf6680c9a90f1612: Status 404 returned error can't find the container with id 8762bcb48db1d504456603795d1a95d28a7ac124c59b7a3dbf6680c9a90f1612
Sep 09 21:13:52.927117 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.927080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" event={"ID":"444081c8-bcb9-49d4-8823-d4e8c0810779","Type":"ContainerStarted","Data":"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"}
Sep 09 21:13:52.927316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.927138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" event={"ID":"444081c8-bcb9-49d4-8823-d4e8c0810779","Type":"ContainerStarted","Data":"8762bcb48db1d504456603795d1a95d28a7ac124c59b7a3dbf6680c9a90f1612"}
Sep 09 21:13:52.927316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.927186    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:13:52.954338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:52.954289    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podStartSLOduration=1.9542749000000001 podStartE2EDuration="1.9542749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:52.953974931 +0000 UTC m=+1249.705519865" watchObservedRunningTime="2026-09-09 21:13:52.9542749 +0000 UTC m=+1249.705819810"
Sep 09 21:13:58.174978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.174952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:13:58.291590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.291480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs\") pod \"e11652f9-f1d0-40c2-b381-9f2890ee9461\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") "
Sep 09 21:13:58.291590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.291542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqj7r\" (UniqueName: \"kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r\") pod \"e11652f9-f1d0-40c2-b381-9f2890ee9461\" (UID: \"e11652f9-f1d0-40c2-b381-9f2890ee9461\") "
Sep 09 21:13:58.293767 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.293739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e11652f9-f1d0-40c2-b381-9f2890ee9461" (UID: "e11652f9-f1d0-40c2-b381-9f2890ee9461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:58.293876 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.293806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r" (OuterVolumeSpecName: "kube-api-access-pqj7r") pod "e11652f9-f1d0-40c2-b381-9f2890ee9461" (UID: "e11652f9-f1d0-40c2-b381-9f2890ee9461"). InnerVolumeSpecName "kube-api-access-pqj7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:58.392892 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.392857    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11652f9-f1d0-40c2-b381-9f2890ee9461-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:13:58.392892 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.392888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqj7r\" (UniqueName: \"kubernetes.io/projected/e11652f9-f1d0-40c2-b381-9f2890ee9461-kube-api-access-pqj7r\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:13:58.954504 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.954456    2563 generic.go:356] "Generic (PLEG): container finished" podID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerID="1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f" exitCode=0
Sep 09 21:13:58.954645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.954521    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"
Sep 09 21:13:58.954645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.954540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" event={"ID":"e11652f9-f1d0-40c2-b381-9f2890ee9461","Type":"ContainerDied","Data":"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"}
Sep 09 21:13:58.954645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.954600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk" event={"ID":"e11652f9-f1d0-40c2-b381-9f2890ee9461","Type":"ContainerDied","Data":"d66d92d817bc4b10b1020c4239291cde446d7f6e0a8798a225d653527f6ed1e0"}
Sep 09 21:13:58.954645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.954630    2563 scope.go:117] "RemoveContainer" containerID="1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"
Sep 09 21:13:58.963643 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.963626    2563 scope.go:117] "RemoveContainer" containerID="1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"
Sep 09 21:13:58.963896 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:13:58.963879    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f\": container with ID starting with 1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f not found: ID does not exist" containerID="1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"
Sep 09 21:13:58.963970 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.963905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f"} err="failed to get container status \"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f\": rpc error: code = NotFound desc = could not find container \"1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f\": container with ID starting with 1f5b3ff77777d41d8c57a6c9c90cdd8932f0fcb1ff1a1cae3e9bbb54c41e288f not found: ID does not exist"
Sep 09 21:13:58.990892 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.990861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:13:58.996077 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:58.996048    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-lqffk"]
Sep 09 21:13:59.715846 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:13:59.715815    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" path="/var/lib/kubelet/pods/e11652f9-f1d0-40c2-b381-9f2890ee9461/volumes"
Sep 09 21:14:11.368643 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.368611    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:14:11.369137 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.369114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main"
Sep 09 21:14:11.369181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.369140    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main"
Sep 09 21:14:11.369229 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.369220    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11652f9-f1d0-40c2-b381-9f2890ee9461" containerName="main"
Sep 09 21:14:11.373476 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.373460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.377559 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.377541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-lpn6n\""
Sep 09 21:14:11.378852 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.378825    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 21:14:11.378973 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.378859    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 21:14:11.378973 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.378904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 21:14:11.379094 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.379002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-qlvtv\""
Sep 09 21:14:11.382666 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.382642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:14:11.507169 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.507134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.507169 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.507172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99l8w\" (UniqueName: \"kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.608280 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.608233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.608280 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.608282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99l8w\" (UniqueName: \"kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.610701 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.610681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.617989 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.617958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99l8w\" (UniqueName: \"kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.684544 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.684450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:11.964602 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:11.964579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:14:11.966599 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:14:11.966570    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e94e705_5c20_4e4a_b16d_f92e79f8eddb.slice/crio-575fdd59b284b67052a28553e3052462b976e7ff19e17c95bc0b7d7f6c567952 WatchSource:0}: Error finding container 575fdd59b284b67052a28553e3052462b976e7ff19e17c95bc0b7d7f6c567952: Status 404 returned error can't find the container with id 575fdd59b284b67052a28553e3052462b976e7ff19e17c95bc0b7d7f6c567952
Sep 09 21:14:12.013304 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:12.013268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" event={"ID":"2e94e705-5c20-4e4a-b16d-f92e79f8eddb","Type":"ContainerStarted","Data":"575fdd59b284b67052a28553e3052462b976e7ff19e17c95bc0b7d7f6c567952"}
Sep 09 21:14:13.019997 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:13.019962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" event={"ID":"2e94e705-5c20-4e4a-b16d-f92e79f8eddb","Type":"ContainerStarted","Data":"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"}
Sep 09 21:14:13.020381 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:13.020048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:23.933387 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:23.933351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:14:23.955097 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:23.955052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podStartSLOduration=12.955036045 podStartE2EDuration="12.955036045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:13.054826173 +0000 UTC m=+1269.806371093" watchObservedRunningTime="2026-09-09 21:14:23.955036045 +0000 UTC m=+1280.706580955"
Sep 09 21:14:25.898888 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:25.898854    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:14:33.932743 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:33.932698    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:14:40.920376 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:40.920281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main" containerID="cri-o://78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214" gracePeriod=45
Sep 09 21:14:43.932732 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:43.932688    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:14:44.026085 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:44.026054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:14:53.933710 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:53.933667    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:14:53.934185 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:14:53.933775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:15:03.933139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:03.933092    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:15:11.160240 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.160211    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:15:11.277360 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.277277    2563 generic.go:356] "Generic (PLEG): container finished" podID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerID="78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214" exitCode=0
Sep 09 21:15:11.277360 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.277322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"
Sep 09 21:15:11.277360 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.277341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" event={"ID":"444081c8-bcb9-49d4-8823-d4e8c0810779","Type":"ContainerDied","Data":"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"}
Sep 09 21:15:11.277580 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.277393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7" event={"ID":"444081c8-bcb9-49d4-8823-d4e8c0810779","Type":"ContainerDied","Data":"8762bcb48db1d504456603795d1a95d28a7ac124c59b7a3dbf6680c9a90f1612"}
Sep 09 21:15:11.277580 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.277417    2563 scope.go:117] "RemoveContainer" containerID="78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"
Sep 09 21:15:11.285685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.285668    2563 scope.go:117] "RemoveContainer" containerID="78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"
Sep 09 21:15:11.285979 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:15:11.285957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214\": container with ID starting with 78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214 not found: ID does not exist" containerID="78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"
Sep 09 21:15:11.286034 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.285988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214"} err="failed to get container status \"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214\": rpc error: code = NotFound desc = could not find container \"78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214\": container with ID starting with 78b933da671d5e08a73386ea7f8656a824a007cfd6717b198df5690959197214 not found: ID does not exist"
Sep 09 21:15:11.292258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.292242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs\") pod \"444081c8-bcb9-49d4-8823-d4e8c0810779\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") "
Sep 09 21:15:11.292345 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.292318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhsfs\" (UniqueName: \"kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs\") pod \"444081c8-bcb9-49d4-8823-d4e8c0810779\" (UID: \"444081c8-bcb9-49d4-8823-d4e8c0810779\") "
Sep 09 21:15:11.294289 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.294268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "444081c8-bcb9-49d4-8823-d4e8c0810779" (UID: "444081c8-bcb9-49d4-8823-d4e8c0810779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:11.294376 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.294314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs" (OuterVolumeSpecName: "kube-api-access-rhsfs") pod "444081c8-bcb9-49d4-8823-d4e8c0810779" (UID: "444081c8-bcb9-49d4-8823-d4e8c0810779"). InnerVolumeSpecName "kube-api-access-rhsfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:11.393090 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.393054    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/444081c8-bcb9-49d4-8823-d4e8c0810779-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:15:11.393090 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.393084    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhsfs\" (UniqueName: \"kubernetes.io/projected/444081c8-bcb9-49d4-8823-d4e8c0810779-kube-api-access-rhsfs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:15:11.607651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.607616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:15:11.613160 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.613134    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7q4b7"]
Sep 09 21:15:11.715052 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:11.715016    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" path="/var/lib/kubelet/pods/444081c8-bcb9-49d4-8823-d4e8c0810779/volumes"
Sep 09 21:15:26.518798 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.518719    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:15:26.519302 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.519275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main"
Sep 09 21:15:26.519302 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.519301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main"
Sep 09 21:15:26.519521 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.519384    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="444081c8-bcb9-49d4-8823-d4e8c0810779" containerName="main"
Sep 09 21:15:26.523801 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.523778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.528213 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.528189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-j4qtt\""
Sep 09 21:15:26.529373 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.529343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 09 21:15:26.529477 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.529342    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-ddrdz\""
Sep 09 21:15:26.529477 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.529349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 09 21:15:26.529477 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.529351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 09 21:15:26.533690 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.533670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:15:26.623060 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.623021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klh7f\" (UniqueName: \"kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.623229 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.623070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.723480 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.723441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klh7f\" (UniqueName: \"kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.723659 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.723627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.726117 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.726091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.733413 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.733388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klh7f\" (UniqueName: \"kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:26.834550 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:26.834438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:27.114645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:27.114529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:15:27.117228 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:15:27.117202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60efccab_1f76_4cfc_aeec_ad8295a47032.slice/crio-b3532dc9503f704550732718f1f3ef486be975227886e3f6bc2ca0e0f1de4492 WatchSource:0}: Error finding container b3532dc9503f704550732718f1f3ef486be975227886e3f6bc2ca0e0f1de4492: Status 404 returned error can't find the container with id b3532dc9503f704550732718f1f3ef486be975227886e3f6bc2ca0e0f1de4492
Sep 09 21:15:27.349347 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:27.349310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" event={"ID":"60efccab-1f76-4cfc-aeec-ad8295a47032","Type":"ContainerStarted","Data":"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"}
Sep 09 21:15:27.349553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:27.349362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" event={"ID":"60efccab-1f76-4cfc-aeec-ad8295a47032","Type":"ContainerStarted","Data":"b3532dc9503f704550732718f1f3ef486be975227886e3f6bc2ca0e0f1de4492"}
Sep 09 21:15:27.349553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:27.349393    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:15:27.372393 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:27.372330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podStartSLOduration=1.37231391 podStartE2EDuration="1.37231391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:27.371183332 +0000 UTC m=+1344.122728237" watchObservedRunningTime="2026-09-09 21:15:27.37231391 +0000 UTC m=+1344.123858818"
Sep 09 21:15:58.355801 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:15:58.355763    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:16:00.026473 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:00.026434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:16:08.355969 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:08.355930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:13.187895 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:13.187865    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:16:14.026047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:14.026000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:15.047530 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:15.047470    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main" containerID="cri-o://77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006" gracePeriod=45
Sep 09 21:16:18.355683 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:18.355641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:24.025701 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:24.025661    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:28.208010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:28.207939    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" containerID="cri-o://fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5" gracePeriod=45
Sep 09 21:16:28.355433 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:28.355395    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:28.355608 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:28.355514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:16:34.025984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:34.025936    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:34.026517 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:34.026056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:16:38.355986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:38.355942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:44.026456 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:44.026409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:45.289039 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.289017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:16:45.434174 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.434143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs\") pod \"60efccab-1f76-4cfc-aeec-ad8295a47032\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") "
Sep 09 21:16:45.434350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.434197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klh7f\" (UniqueName: \"kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f\") pod \"60efccab-1f76-4cfc-aeec-ad8295a47032\" (UID: \"60efccab-1f76-4cfc-aeec-ad8295a47032\") "
Sep 09 21:16:45.436306 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.436279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60efccab-1f76-4cfc-aeec-ad8295a47032" (UID: "60efccab-1f76-4cfc-aeec-ad8295a47032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:45.436432 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.436309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f" (OuterVolumeSpecName: "kube-api-access-klh7f") pod "60efccab-1f76-4cfc-aeec-ad8295a47032" (UID: "60efccab-1f76-4cfc-aeec-ad8295a47032"). InnerVolumeSpecName "kube-api-access-klh7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:45.535584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.535559    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klh7f\" (UniqueName: \"kubernetes.io/projected/60efccab-1f76-4cfc-aeec-ad8295a47032-kube-api-access-klh7f\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.535584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.535581    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efccab-1f76-4cfc-aeec-ad8295a47032-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.695350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.695278    2563 generic.go:356] "Generic (PLEG): container finished" podID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerID="77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006" exitCode=0
Sep 09 21:16:45.695350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.695323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"
Sep 09 21:16:45.695350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.695340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" event={"ID":"60efccab-1f76-4cfc-aeec-ad8295a47032","Type":"ContainerDied","Data":"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"}
Sep 09 21:16:45.695577 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.695383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv" event={"ID":"60efccab-1f76-4cfc-aeec-ad8295a47032","Type":"ContainerDied","Data":"b3532dc9503f704550732718f1f3ef486be975227886e3f6bc2ca0e0f1de4492"}
Sep 09 21:16:45.695577 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.695407    2563 scope.go:117] "RemoveContainer" containerID="77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"
Sep 09 21:16:45.704500 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.704468    2563 scope.go:117] "RemoveContainer" containerID="77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"
Sep 09 21:16:45.704848 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:16:45.704828    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006\": container with ID starting with 77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006 not found: ID does not exist" containerID="77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"
Sep 09 21:16:45.704941 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.704855    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006"} err="failed to get container status \"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006\": rpc error: code = NotFound desc = could not find container \"77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006\": container with ID starting with 77fa4e9debebbb91cb9c15608e0322de8eb413c2d86f7b892c181114bd24e006 not found: ID does not exist"
Sep 09 21:16:45.720762 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.720727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:16:45.722232 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:45.722215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9krtmv"]
Sep 09 21:16:47.715153 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:47.715120    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" path="/var/lib/kubelet/pods/60efccab-1f76-4cfc-aeec-ad8295a47032/volumes"
Sep 09 21:16:54.026315 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.026270    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:54.074077 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.074043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:16:54.074425 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.074411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main"
Sep 09 21:16:54.074510 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.074427    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main"
Sep 09 21:16:54.074510 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.074497    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60efccab-1f76-4cfc-aeec-ad8295a47032" containerName="main"
Sep 09 21:16:54.078789 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.078767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.084522 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.084471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-4n9lv\""
Sep 09 21:16:54.084639 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.084534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 09 21:16:54.084639 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.084550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9nq4p\""
Sep 09 21:16:54.084639 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.084615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 09 21:16:54.084639 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.084471    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 09 21:16:54.094823 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.094800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:16:54.209963 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.209932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlq8\" (UniqueName: \"kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.210119 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.209991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.310421 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.310348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlq8\" (UniqueName: \"kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.310421 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.310413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.312834 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.312811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.321097 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.321067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlq8\" (UniqueName: \"kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.389538 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.389515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:54.673288 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.673257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:16:54.676575 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:16:54.676546    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dfafb5_3c4e_4cfc_98e2_8950382dc2f9.slice/crio-0e29792a7910a5d792d78b102063e7617566f1dc13210978b086b3b40617b7da WatchSource:0}: Error finding container 0e29792a7910a5d792d78b102063e7617566f1dc13210978b086b3b40617b7da: Status 404 returned error can't find the container with id 0e29792a7910a5d792d78b102063e7617566f1dc13210978b086b3b40617b7da
Sep 09 21:16:54.736938 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:54.736911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" event={"ID":"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9","Type":"ContainerStarted","Data":"0e29792a7910a5d792d78b102063e7617566f1dc13210978b086b3b40617b7da"}
Sep 09 21:16:55.742690 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:55.742652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" event={"ID":"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9","Type":"ContainerStarted","Data":"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"}
Sep 09 21:16:55.743152 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:55.742784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:16:55.763258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:55.763211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" podStartSLOduration=1.7631977239999999 podStartE2EDuration="1.763197724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:55.762098117 +0000 UTC m=+1432.513643026" watchObservedRunningTime="2026-09-09 21:16:55.763197724 +0000 UTC m=+1432.514742649"
Sep 09 21:16:58.458953 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.458930    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:16:58.544238 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.544176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99l8w\" (UniqueName: \"kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w\") pod \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") "
Sep 09 21:16:58.544357 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.544279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs\") pod \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\" (UID: \"2e94e705-5c20-4e4a-b16d-f92e79f8eddb\") "
Sep 09 21:16:58.546314 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.546291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e94e705-5c20-4e4a-b16d-f92e79f8eddb" (UID: "2e94e705-5c20-4e4a-b16d-f92e79f8eddb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:58.546454 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.546434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w" (OuterVolumeSpecName: "kube-api-access-99l8w") pod "2e94e705-5c20-4e4a-b16d-f92e79f8eddb" (UID: "2e94e705-5c20-4e4a-b16d-f92e79f8eddb"). InnerVolumeSpecName "kube-api-access-99l8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:58.645388 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.645359    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:16:58.645388 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.645385    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99l8w\" (UniqueName: \"kubernetes.io/projected/2e94e705-5c20-4e4a-b16d-f92e79f8eddb-kube-api-access-99l8w\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:16:58.757590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.757553    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerID="fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5" exitCode=0
Sep 09 21:16:58.757751 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.757609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"
Sep 09 21:16:58.757751 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.757617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" event={"ID":"2e94e705-5c20-4e4a-b16d-f92e79f8eddb","Type":"ContainerDied","Data":"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"}
Sep 09 21:16:58.757751 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.757680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h" event={"ID":"2e94e705-5c20-4e4a-b16d-f92e79f8eddb","Type":"ContainerDied","Data":"575fdd59b284b67052a28553e3052462b976e7ff19e17c95bc0b7d7f6c567952"}
Sep 09 21:16:58.757751 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.757712    2563 scope.go:117] "RemoveContainer" containerID="fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"
Sep 09 21:16:58.773012 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.772990    2563 scope.go:117] "RemoveContainer" containerID="fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"
Sep 09 21:16:58.773271 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:16:58.773246    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5\": container with ID starting with fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5 not found: ID does not exist" containerID="fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"
Sep 09 21:16:58.773351 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.773281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5"} err="failed to get container status \"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5\": rpc error: code = NotFound desc = could not find container \"fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5\": container with ID starting with fc8f27b67f7757283b0b234bafa8b75c8b9288799fd652866a311b0a084c2df5 not found: ID does not exist"
Sep 09 21:16:58.794665 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.794607    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:16:58.801102 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:58.801082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c5vz8h"]
Sep 09 21:16:59.715599 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:16:59.715566    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" path="/var/lib/kubelet/pods/2e94e705-5c20-4e4a-b16d-f92e79f8eddb/volumes"
Sep 09 21:17:11.348921 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.348891    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:17:11.349338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.349243    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main"
Sep 09 21:17:11.349338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.349257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main"
Sep 09 21:17:11.349338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.349324    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e94e705-5c20-4e4a-b16d-f92e79f8eddb" containerName="main"
Sep 09 21:17:11.352283 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.352260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.356622 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.356595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-2mth8\""
Sep 09 21:17:11.356758 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.356620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 09 21:17:11.356758 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.356666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-6g76z\""
Sep 09 21:17:11.356758 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.356694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 09 21:17:11.356930 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.356916    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 09 21:17:11.362713 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.362689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:17:11.451994 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.451964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lxd\" (UniqueName: \"kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.452160 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.452000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.552832 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.552795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lxd\" (UniqueName: \"kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.553016 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.552842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.555367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.555341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.563817 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.563791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lxd\" (UniqueName: \"kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.663068 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.663035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:11.934585 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:11.934510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:17:11.937422 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:17:11.937393    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cab1d5b_1c18_4ea5_b97b_bc5d046962b1.slice/crio-a61b4345b1546bef53fbd7b9a799eee273316440fee782482739c049b81b645e WatchSource:0}: Error finding container a61b4345b1546bef53fbd7b9a799eee273316440fee782482739c049b81b645e: Status 404 returned error can't find the container with id a61b4345b1546bef53fbd7b9a799eee273316440fee782482739c049b81b645e
Sep 09 21:17:12.822343 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:12.822303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" event={"ID":"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1","Type":"ContainerStarted","Data":"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"}
Sep 09 21:17:12.822830 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:12.822379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" event={"ID":"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1","Type":"ContainerStarted","Data":"a61b4345b1546bef53fbd7b9a799eee273316440fee782482739c049b81b645e"}
Sep 09 21:17:12.822830 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:12.822402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:17:12.853162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:12.853112    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podStartSLOduration=1.853097212 podStartE2EDuration="1.853097212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:17:12.852409169 +0000 UTC m=+1449.603954097" watchObservedRunningTime="2026-09-09 21:17:12.853097212 +0000 UTC m=+1449.604642120"
Sep 09 21:17:26.748473 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:26.748443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:17:28.268755 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:28.268709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:17:36.749368 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:36.749324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:17:39.008175 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.008141    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 09 21:17:39.008427 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:17:39.008176    2563 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main" containerID="cri-o://f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"
Sep 09 21:17:39.008427 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.008218    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main" containerID="cri-o://f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042" gracePeriod=50
Sep 09 21:17:39.147180 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.147156    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs_94dfafb5-3c4e-4cfc-98e2-8950382dc2f9/main/0.log"
Sep 09 21:17:39.149378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.149362    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:17:39.284590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.284441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqlq8\" (UniqueName: \"kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8\") pod \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") "
Sep 09 21:17:39.284590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.284477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs\") pod \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\" (UID: \"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9\") "
Sep 09 21:17:39.286571 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.286540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" (UID: "94dfafb5-3c4e-4cfc-98e2-8950382dc2f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:17:39.286685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.286588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8" (OuterVolumeSpecName: "kube-api-access-zqlq8") pod "94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" (UID: "94dfafb5-3c4e-4cfc-98e2-8950382dc2f9"). InnerVolumeSpecName "kube-api-access-zqlq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:17:39.386065 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.386038    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlq8\" (UniqueName: \"kubernetes.io/projected/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-kube-api-access-zqlq8\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.386065 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.386061    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:17:39.944194 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944165    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs_94dfafb5-3c4e-4cfc-98e2-8950382dc2f9/main/0.log"
Sep 09 21:17:39.944364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944209    2563 generic.go:356] "Generic (PLEG): container finished" podID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerID="f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042" exitCode=1
Sep 09 21:17:39.944364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"
Sep 09 21:17:39.944364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" event={"ID":"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9","Type":"ContainerDied","Data":"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"}
Sep 09 21:17:39.944364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs" event={"ID":"94dfafb5-3c4e-4cfc-98e2-8950382dc2f9","Type":"ContainerDied","Data":"0e29792a7910a5d792d78b102063e7617566f1dc13210978b086b3b40617b7da"}
Sep 09 21:17:39.944364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.944363    2563 scope.go:117] "RemoveContainer" containerID="f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"
Sep 09 21:17:39.952388 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.952370    2563 scope.go:117] "RemoveContainer" containerID="f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"
Sep 09 21:17:39.952669 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:17:39.952648    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042\": container with ID starting with f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042 not found: ID does not exist" containerID="f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"
Sep 09 21:17:39.952739 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.952682    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042"} err="failed to get container status \"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042\": rpc error: code = NotFound desc = could not find container \"f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042\": container with ID starting with f6f1576bf44f41bf859126f41ad4dd23797b9bd9bbdc46417611266a4587a042 not found: ID does not exist"
Sep 09 21:17:39.965671 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.965645    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:17:39.972123 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:39.972104    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c2mjs"]
Sep 09 21:17:41.715129 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:41.715093    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" path="/var/lib/kubelet/pods/94dfafb5-3c4e-4cfc-98e2-8950382dc2f9/volumes"
Sep 09 21:17:43.829852 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:17:43.829824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:18:03.735812 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:18:03.735785    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:18:03.738278 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:18:03.738255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:19:16.650089 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:16.650053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:19:23.829986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:23.829952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:31.669424 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:31.669381    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main" containerID="cri-o://540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c" gracePeriod=45
Sep 09 21:19:33.830632 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:33.830590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:43.830157 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:43.830120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:43.830576 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:43.830236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:19:53.829881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:19:53.829845    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:20:01.923841 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:01.923810    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:20:01.939872 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:01.939783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs\") pod \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") "
Sep 09 21:20:01.939872 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:01.939831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4lxd\" (UniqueName: \"kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd\") pod \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\" (UID: \"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1\") "
Sep 09 21:20:01.942004 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:01.941973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd" (OuterVolumeSpecName: "kube-api-access-q4lxd") pod "9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" (UID: "9cab1d5b-1c18-4ea5-b97b-bc5d046962b1"). InnerVolumeSpecName "kube-api-access-q4lxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:01.942125 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:01.942008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" (UID: "9cab1d5b-1c18-4ea5-b97b-bc5d046962b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:02.040700 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.040656    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:20:02.040700 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.040693    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4lxd\" (UniqueName: \"kubernetes.io/projected/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1-kube-api-access-q4lxd\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:20:02.579043 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.579008    2563 generic.go:356] "Generic (PLEG): container finished" podID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerID="540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c" exitCode=0
Sep 09 21:20:02.579226 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.579049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"
Sep 09 21:20:02.579226 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.579093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" event={"ID":"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1","Type":"ContainerDied","Data":"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"}
Sep 09 21:20:02.579226 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.579142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs" event={"ID":"9cab1d5b-1c18-4ea5-b97b-bc5d046962b1","Type":"ContainerDied","Data":"a61b4345b1546bef53fbd7b9a799eee273316440fee782482739c049b81b645e"}
Sep 09 21:20:02.579226 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.579166    2563 scope.go:117] "RemoveContainer" containerID="540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"
Sep 09 21:20:02.588034 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.588020    2563 scope.go:117] "RemoveContainer" containerID="540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"
Sep 09 21:20:02.588284 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:20:02.588264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c\": container with ID starting with 540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c not found: ID does not exist" containerID="540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"
Sep 09 21:20:02.588333 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.588292    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c"} err="failed to get container status \"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c\": rpc error: code = NotFound desc = could not find container \"540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c\": container with ID starting with 540e28777574714f92c9961af9afb936f23e53527d1c01e94063a8cfb072533c not found: ID does not exist"
Sep 09 21:20:02.603318 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.603287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:20:02.609272 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:02.609251    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wmcjs"]
Sep 09 21:20:03.715438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:03.715408    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" path="/var/lib/kubelet/pods/9cab1d5b-1c18-4ea5-b97b-bc5d046962b1/volumes"
Sep 09 21:20:09.205098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205058    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:20:09.205501 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main"
Sep 09 21:20:09.205501 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main"
Sep 09 21:20:09.205501 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205437    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main"
Sep 09 21:20:09.205501 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main"
Sep 09 21:20:09.205673 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205513    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cab1d5b-1c18-4ea5-b97b-bc5d046962b1" containerName="main"
Sep 09 21:20:09.205673 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.205524    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94dfafb5-3c4e-4cfc-98e2-8950382dc2f9" containerName="main"
Sep 09 21:20:09.208098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.208080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.212318 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.212293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-bvddm\""
Sep 09 21:20:09.212449 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.212334    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 21:20:09.212533 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.212465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6bqsz\""
Sep 09 21:20:09.212620 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.212305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 21:20:09.212620 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.212636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 21:20:09.230631 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.230606    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:20:09.309654 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.309623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck2s\" (UniqueName: \"kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.309654 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.309658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.410927 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.410896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck2s\" (UniqueName: \"kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.411068 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.410934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.413452 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.413418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.419328 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.419304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck2s\" (UniqueName: \"kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.520733 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.520672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:09.807765 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:09.807690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:20:09.811436 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:20:09.811406    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93007eda_f22b_463f_92ab_fc30c90f6de7.slice/crio-5691c339b6d85973fe3581b098eb89691f002935bb57a5af59584aa6121debe6 WatchSource:0}: Error finding container 5691c339b6d85973fe3581b098eb89691f002935bb57a5af59584aa6121debe6: Status 404 returned error can't find the container with id 5691c339b6d85973fe3581b098eb89691f002935bb57a5af59584aa6121debe6
Sep 09 21:20:10.617026 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:10.616984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" event={"ID":"93007eda-f22b-463f-92ab-fc30c90f6de7","Type":"ContainerStarted","Data":"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"}
Sep 09 21:20:10.617427 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:10.617057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" event={"ID":"93007eda-f22b-463f-92ab-fc30c90f6de7","Type":"ContainerStarted","Data":"5691c339b6d85973fe3581b098eb89691f002935bb57a5af59584aa6121debe6"}
Sep 09 21:20:10.617427 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:10.617089    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:20:10.635546 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:10.635470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podStartSLOduration=1.635454162 podStartE2EDuration="1.635454162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:10.635244055 +0000 UTC m=+1627.386788990" watchObservedRunningTime="2026-09-09 21:20:10.635454162 +0000 UTC m=+1627.386999083"
Sep 09 21:20:41.623139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:20:41.623107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:21:59.638727 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:21:59.638695    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:22:01.623573 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:01.623528    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:11.623304 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:11.623270    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:14.659325 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:14.659279    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" containerID="cri-o://cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904" gracePeriod=45
Sep 09 21:22:21.622562 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:21.622519    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:21.622973 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:21.622641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:22:31.622618 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:31.622576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:41.622671 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:41.622623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:44.911126 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:44.911101    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:22:44.976735 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:44.976674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs\") pod \"93007eda-f22b-463f-92ab-fc30c90f6de7\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") "
Sep 09 21:22:44.976735 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:44.976709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lck2s\" (UniqueName: \"kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s\") pod \"93007eda-f22b-463f-92ab-fc30c90f6de7\" (UID: \"93007eda-f22b-463f-92ab-fc30c90f6de7\") "
Sep 09 21:22:44.978732 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:44.978700    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93007eda-f22b-463f-92ab-fc30c90f6de7" (UID: "93007eda-f22b-463f-92ab-fc30c90f6de7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:44.978825 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:44.978777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s" (OuterVolumeSpecName: "kube-api-access-lck2s") pod "93007eda-f22b-463f-92ab-fc30c90f6de7" (UID: "93007eda-f22b-463f-92ab-fc30c90f6de7"). InnerVolumeSpecName "kube-api-access-lck2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:45.078069 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.078026    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93007eda-f22b-463f-92ab-fc30c90f6de7-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.078069 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.078065    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck2s\" (UniqueName: \"kubernetes.io/projected/93007eda-f22b-463f-92ab-fc30c90f6de7-kube-api-access-lck2s\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.298987 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.298911    2563 generic.go:356] "Generic (PLEG): container finished" podID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerID="cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904" exitCode=0
Sep 09 21:22:45.298987 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.298954    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"
Sep 09 21:22:45.299162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.298997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" event={"ID":"93007eda-f22b-463f-92ab-fc30c90f6de7","Type":"ContainerDied","Data":"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"}
Sep 09 21:22:45.299162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.299054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z" event={"ID":"93007eda-f22b-463f-92ab-fc30c90f6de7","Type":"ContainerDied","Data":"5691c339b6d85973fe3581b098eb89691f002935bb57a5af59584aa6121debe6"}
Sep 09 21:22:45.299162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.299083    2563 scope.go:117] "RemoveContainer" containerID="cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"
Sep 09 21:22:45.308066 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.308046    2563 scope.go:117] "RemoveContainer" containerID="cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"
Sep 09 21:22:45.308341 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:22:45.308323    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904\": container with ID starting with cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904 not found: ID does not exist" containerID="cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"
Sep 09 21:22:45.308406 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.308350    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904"} err="failed to get container status \"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904\": rpc error: code = NotFound desc = could not find container \"cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904\": container with ID starting with cc8d9c911b733bb9a4f60676c65d0c2d819afebde34726b7b578b71525b73904 not found: ID does not exist"
Sep 09 21:22:45.324160 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.324135    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:22:45.331452 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.331432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75j526z"]
Sep 09 21:22:45.715013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:22:45.714980    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" path="/var/lib/kubelet/pods/93007eda-f22b-463f-92ab-fc30c90f6de7/volumes"
Sep 09 21:23:03.760936 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:03.760904    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:23:03.764045 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:03.764020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:23:15.363633 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.363598    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:23:15.364161 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.364142    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main"
Sep 09 21:23:15.364236 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.364165    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main"
Sep 09 21:23:15.364291 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.364276    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93007eda-f22b-463f-92ab-fc30c90f6de7" containerName="main"
Sep 09 21:23:15.367247 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.367225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.372341 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.372319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 21:23:15.372474 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.372461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-lqht4\""
Sep 09 21:23:15.372586 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.372555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-xp42k\""
Sep 09 21:23:15.372670 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.372558    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 21:23:15.372670 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.372561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 21:23:15.387381 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.387362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:23:15.525553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.525519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.525716 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.525575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7r4\" (UniqueName: \"kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.626216 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.626184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.626392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.626238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7r4\" (UniqueName: \"kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.628753 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.628729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.640981 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.640952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7r4\" (UniqueName: \"kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.677113 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.677087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:15.951102 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:15.951020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:23:15.954261 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:23:15.954233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dda7db_51b3_41d7_a4a0_0e469cccec2c.slice/crio-1e31db4e1ba9231c067c0b4b152218838987d25c30797ade0aa7cdcc230fb85b WatchSource:0}: Error finding container 1e31db4e1ba9231c067c0b4b152218838987d25c30797ade0aa7cdcc230fb85b: Status 404 returned error can't find the container with id 1e31db4e1ba9231c067c0b4b152218838987d25c30797ade0aa7cdcc230fb85b
Sep 09 21:23:16.445336 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:16.445295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" event={"ID":"70dda7db-51b3-41d7-a4a0-0e469cccec2c","Type":"ContainerStarted","Data":"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"}
Sep 09 21:23:16.445824 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:16.445357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" event={"ID":"70dda7db-51b3-41d7-a4a0-0e469cccec2c","Type":"ContainerStarted","Data":"1e31db4e1ba9231c067c0b4b152218838987d25c30797ade0aa7cdcc230fb85b"}
Sep 09 21:23:16.445824 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:16.445417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:16.469180 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:16.469134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podStartSLOduration=1.4691189740000001 podStartE2EDuration="1.469118974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:16.467254942 +0000 UTC m=+1813.218799853" watchObservedRunningTime="2026-09-09 21:23:16.469118974 +0000 UTC m=+1813.220663884"
Sep 09 21:23:38.286750 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.286712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:23:38.289883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.289863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.292875 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.292849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 09 21:23:38.293009 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.292913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 09 21:23:38.294305 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.294283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 21:23:38.294424 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.294386    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-q7vgc\""
Sep 09 21:23:38.309212 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.309187    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:23:38.310968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.310947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.311070 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.310983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.311070 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.311024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.311186 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.311097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.311186 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.311141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b8nq\" (UniqueName: \"kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.311281 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.311183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411596 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b8nq\" (UniqueName: \"kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411803 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411860 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.411934 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.411908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.412045 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.412020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.412287 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.412266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.412287 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.412263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.414462 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.414434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.414596 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.414499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.420319 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.420296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b8nq\" (UniqueName: \"kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq\") pod \"test-div-v1-kserve-fb566d667-2wqsf\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.601442 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.601373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:38.879422 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.879398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:23:38.881608 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:23:38.881581    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7e1d6b_07f5_4c6e_8221_893c09abc898.slice/crio-aee9d8046f9affe15e1fac6dc5962806abee2503345df1f93d6bbe59dd9cba20 WatchSource:0}: Error finding container aee9d8046f9affe15e1fac6dc5962806abee2503345df1f93d6bbe59dd9cba20: Status 404 returned error can't find the container with id aee9d8046f9affe15e1fac6dc5962806abee2503345df1f93d6bbe59dd9cba20
Sep 09 21:23:38.883283 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:38.883269    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:23:39.549960 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:39.549917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" event={"ID":"cd7e1d6b-07f5-4c6e-8221-893c09abc898","Type":"ContainerStarted","Data":"aee9d8046f9affe15e1fac6dc5962806abee2503345df1f93d6bbe59dd9cba20"}
Sep 09 21:23:40.556050 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:40.556012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" event={"ID":"cd7e1d6b-07f5-4c6e-8221-893c09abc898","Type":"ContainerStarted","Data":"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"}
Sep 09 21:23:40.581204 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:40.581156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" podStartSLOduration=1.603410257 podStartE2EDuration="2.581141041s" totalImagesPullingTime="977.730784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:40.579205146 +0000 UTC m=+1837.330750052" watchObservedRunningTime="2026-09-09 21:23:40.581141041 +0000 UTC m=+1837.332685950"
Sep 09 21:23:40.601737 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:40.601712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:47.451300 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:47.451271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:23:48.082778 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.082744    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:23:48.086001 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.085979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.089714 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.089695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 21:23:48.099438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.099415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:23:48.195371 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.195522 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.195522 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.195522 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.195657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.195657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.195589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqm6t\" (UniqueName: \"kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.295988 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.295956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqm6t\" (UniqueName: \"kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296477 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.296590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.296572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.298327 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.298304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.298640 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.298622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.306546 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.306527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqm6t\" (UniqueName: \"kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t\") pod \"test-div-v2-kserve-5f6fdb6b77-xm7sr\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.396844 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.396805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:48.601534 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.601472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:48.614213 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.614191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:48.671338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:48.671276    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:23:48.674585 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:23:48.674555    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b918f82_ecf1_4724_9e3d_5a44d8515f4f.slice/crio-51215c8a757443da695459818287decddf5cbc88e6e663bf722893c1887753da WatchSource:0}: Error finding container 51215c8a757443da695459818287decddf5cbc88e6e663bf722893c1887753da: Status 404 returned error can't find the container with id 51215c8a757443da695459818287decddf5cbc88e6e663bf722893c1887753da
Sep 09 21:23:49.597264 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:49.597222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" event={"ID":"2b918f82-ecf1-4724-9e3d-5a44d8515f4f","Type":"ContainerStarted","Data":"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"}
Sep 09 21:23:49.597449 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:49.597314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" event={"ID":"2b918f82-ecf1-4724-9e3d-5a44d8515f4f","Type":"ContainerStarted","Data":"51215c8a757443da695459818287decddf5cbc88e6e663bf722893c1887753da"}
Sep 09 21:23:49.608685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:49.608654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:23:49.623954 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:49.623903    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" podStartSLOduration=1.623884704 podStartE2EDuration="1.623884704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:49.621467695 +0000 UTC m=+1846.373012616" watchObservedRunningTime="2026-09-09 21:23:49.623884704 +0000 UTC m=+1846.375429613"
Sep 09 21:23:50.397566 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:50.397530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:58.397589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:58.397550    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:58.409548 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:58.409525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:23:58.649773 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:23:58.649699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:24:10.961310 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:10.961275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:24:17.451575 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:17.451531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:20.120373 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:20.120336    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:24:20.772239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:20.772200    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:24:25.980914 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:25.980826    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main" containerID="cri-o://2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341" gracePeriod=45
Sep 09 21:24:27.451538 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:27.451477    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:35.140647 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.140599    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" podUID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" containerName="main" containerID="cri-o://9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1" gracePeriod=45
Sep 09 21:24:35.390915 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.390858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:24:35.492563 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b8nq\" (UniqueName: \"kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs\") pod \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\" (UID: \"cd7e1d6b-07f5-4c6e-8221-893c09abc898\") "
Sep 09 21:24:35.492978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:35.492978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:35.492978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.492932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home" (OuterVolumeSpecName: "home") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:35.494946 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.494914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq" (OuterVolumeSpecName: "kube-api-access-8b8nq") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "kube-api-access-8b8nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:35.494946 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.494936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:35.495077 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.494978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm" (OuterVolumeSpecName: "dshm") pod "cd7e1d6b-07f5-4c6e-8221-893c09abc898" (UID: "cd7e1d6b-07f5-4c6e-8221-893c09abc898"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:35.593699 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593670    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b8nq\" (UniqueName: \"kubernetes.io/projected/cd7e1d6b-07f5-4c6e-8221-893c09abc898-kube-api-access-8b8nq\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.593699 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593696    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.593838 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593706    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.593838 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593713    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.593838 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593721    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.593838 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.593728    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7e1d6b-07f5-4c6e-8221-893c09abc898-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:35.792371 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.792320    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" podUID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" containerName="main" containerID="cri-o://55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297" gracePeriod=45
Sep 09 21:24:35.815844 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.815814    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" containerID="9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1" exitCode=0
Sep 09 21:24:35.815937 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.815856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"
Sep 09 21:24:35.815937 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.815887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" event={"ID":"cd7e1d6b-07f5-4c6e-8221-893c09abc898","Type":"ContainerDied","Data":"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"}
Sep 09 21:24:35.816017 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.815943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf" event={"ID":"cd7e1d6b-07f5-4c6e-8221-893c09abc898","Type":"ContainerDied","Data":"aee9d8046f9affe15e1fac6dc5962806abee2503345df1f93d6bbe59dd9cba20"}
Sep 09 21:24:35.816017 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.815975    2563 scope.go:117] "RemoveContainer" containerID="9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"
Sep 09 21:24:35.863471 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.863444    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:24:35.867207 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.867185    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2wqsf"]
Sep 09 21:24:35.902589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.902568    2563 scope.go:117] "RemoveContainer" containerID="9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"
Sep 09 21:24:35.902912 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:24:35.902891    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1\": container with ID starting with 9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1 not found: ID does not exist" containerID="9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"
Sep 09 21:24:35.902964 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:35.902923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1"} err="failed to get container status \"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1\": rpc error: code = NotFound desc = could not find container \"9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1\": container with ID starting with 9a2a3819f4e799eaa9d782761f59f36374b8de496ff663d086c6c041283fd4b1 not found: ID does not exist"
Sep 09 21:24:36.034561 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.034538    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:24:36.098279 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqm6t\" (UniqueName: \"kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098429 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098429 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098429 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098429 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098613 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs\") pod \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\" (UID: \"2b918f82-ecf1-4724-9e3d-5a44d8515f4f\") "
Sep 09 21:24:36.098613 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098549    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home" (OuterVolumeSpecName: "home") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:36.098715 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:36.098715 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:36.098808 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098746    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.098808 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098757    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.098808 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.098766    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.100394 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.100371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm" (OuterVolumeSpecName: "dshm") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:36.100516 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.100389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:36.100516 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.100402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t" (OuterVolumeSpecName: "kube-api-access-tqm6t") pod "2b918f82-ecf1-4724-9e3d-5a44d8515f4f" (UID: "2b918f82-ecf1-4724-9e3d-5a44d8515f4f"). InnerVolumeSpecName "kube-api-access-tqm6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:36.199669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.199641    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.199669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.199668    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqm6t\" (UniqueName: \"kubernetes.io/projected/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-kube-api-access-tqm6t\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.200007 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.199676    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b918f82-ecf1-4724-9e3d-5a44d8515f4f-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:36.823016 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.822973    2563 generic.go:356] "Generic (PLEG): container finished" podID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" containerID="55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297" exitCode=0
Sep 09 21:24:36.823220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.823034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"
Sep 09 21:24:36.823220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.823049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" event={"ID":"2b918f82-ecf1-4724-9e3d-5a44d8515f4f","Type":"ContainerDied","Data":"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"}
Sep 09 21:24:36.823220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.823096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr" event={"ID":"2b918f82-ecf1-4724-9e3d-5a44d8515f4f","Type":"ContainerDied","Data":"51215c8a757443da695459818287decddf5cbc88e6e663bf722893c1887753da"}
Sep 09 21:24:36.823220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.823120    2563 scope.go:117] "RemoveContainer" containerID="55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"
Sep 09 21:24:36.832695 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.832675    2563 scope.go:117] "RemoveContainer" containerID="55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"
Sep 09 21:24:36.832947 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:24:36.832927    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297\": container with ID starting with 55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297 not found: ID does not exist" containerID="55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"
Sep 09 21:24:36.833009 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.832958    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297"} err="failed to get container status \"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297\": rpc error: code = NotFound desc = could not find container \"55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297\": container with ID starting with 55e15b0d093651b292a88aaa125ecd08c08944c5d83dd9c301f6cc12ac1d8297 not found: ID does not exist"
Sep 09 21:24:36.849173 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.849147    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:24:36.855881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:36.855860    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xm7sr"]
Sep 09 21:24:37.451570 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:37.451534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:37.451982 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:37.451639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:24:37.716248 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:37.716167    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" path="/var/lib/kubelet/pods/2b918f82-ecf1-4724-9e3d-5a44d8515f4f/volumes"
Sep 09 21:24:37.716578 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:37.716564    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" path="/var/lib/kubelet/pods/cd7e1d6b-07f5-4c6e-8221-893c09abc898/volumes"
Sep 09 21:24:47.450917 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:47.450876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:56.223291 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.223266    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:24:56.274730 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.274659    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs\") pod \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") "
Sep 09 21:24:56.274867 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.274754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw7r4\" (UniqueName: \"kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4\") pod \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\" (UID: \"70dda7db-51b3-41d7-a4a0-0e469cccec2c\") "
Sep 09 21:24:56.279527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.277747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70dda7db-51b3-41d7-a4a0-0e469cccec2c" (UID: "70dda7db-51b3-41d7-a4a0-0e469cccec2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:56.283008 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.282978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4" (OuterVolumeSpecName: "kube-api-access-dw7r4") pod "70dda7db-51b3-41d7-a4a0-0e469cccec2c" (UID: "70dda7db-51b3-41d7-a4a0-0e469cccec2c"). InnerVolumeSpecName "kube-api-access-dw7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:56.376378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.376347    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw7r4\" (UniqueName: \"kubernetes.io/projected/70dda7db-51b3-41d7-a4a0-0e469cccec2c-kube-api-access-dw7r4\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.376378 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.376374    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70dda7db-51b3-41d7-a4a0-0e469cccec2c-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.912333 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.912295    2563 generic.go:356] "Generic (PLEG): container finished" podID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerID="2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341" exitCode=0
Sep 09 21:24:56.912542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.912359    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"
Sep 09 21:24:56.912542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.912360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" event={"ID":"70dda7db-51b3-41d7-a4a0-0e469cccec2c","Type":"ContainerDied","Data":"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"}
Sep 09 21:24:56.912542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.912475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8" event={"ID":"70dda7db-51b3-41d7-a4a0-0e469cccec2c","Type":"ContainerDied","Data":"1e31db4e1ba9231c067c0b4b152218838987d25c30797ade0aa7cdcc230fb85b"}
Sep 09 21:24:56.912542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.912518    2563 scope.go:117] "RemoveContainer" containerID="2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"
Sep 09 21:24:56.921321 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.921300    2563 scope.go:117] "RemoveContainer" containerID="2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"
Sep 09 21:24:56.921612 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:24:56.921593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341\": container with ID starting with 2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341 not found: ID does not exist" containerID="2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"
Sep 09 21:24:56.921680 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.921620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341"} err="failed to get container status \"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341\": rpc error: code = NotFound desc = could not find container \"2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341\": container with ID starting with 2dc89eea3f8cec91f2d1774745b52d09336e7c380ff03fe0db9631cbec0c5341 not found: ID does not exist"
Sep 09 21:24:56.937544 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.937524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:24:56.940300 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:56.940281    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7wnl8"]
Sep 09 21:24:57.715848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:24:57.715811    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" path="/var/lib/kubelet/pods/70dda7db-51b3-41d7-a4a0-0e469cccec2c/volumes"
Sep 09 21:25:04.589942 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.589908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590225    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590237    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590246    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590251    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590271    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590277    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590355    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7e1d6b-07f5-4c6e-8221-893c09abc898" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590364    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b918f82-ecf1-4724-9e3d-5a44d8515f4f" containerName="main"
Sep 09 21:25:04.590367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.590370    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70dda7db-51b3-41d7-a4a0-0e469cccec2c" containerName="main"
Sep 09 21:25:04.594904 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.594886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.597810 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.597789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 09 21:25:04.599080 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.599059    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 09 21:25:04.599080 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.599069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-h5q9x\""
Sep 09 21:25:04.599250 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.599066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 09 21:25:04.604113 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.604089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:25:04.746791 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.746975 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.746975 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.746975 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.746975 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.747204 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.746979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kvz\" (UniqueName: \"kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.847932 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.847845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.847932 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.847889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kvz\" (UniqueName: \"kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.847939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.847959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.847978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848139 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.848000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848374 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.848299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848453 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.848431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.848545 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.848459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.850124 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.850103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.850498 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.850460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.862919 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.862894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kvz\" (UniqueName: \"kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz\") pod \"stop-v2-kserve-7f68747c45-5bntr\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:04.905718 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:04.905692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:05.181722 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:05.181693    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:25:05.184324 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:25:05.184292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6844d54_bda4_46e7_8add_6f072c892ed1.slice/crio-5ac38a12398cab2a7dbff55c012af1c3f4dd9120d6f4df34ba442dc541fd8bde WatchSource:0}: Error finding container 5ac38a12398cab2a7dbff55c012af1c3f4dd9120d6f4df34ba442dc541fd8bde: Status 404 returned error can't find the container with id 5ac38a12398cab2a7dbff55c012af1c3f4dd9120d6f4df34ba442dc541fd8bde
Sep 09 21:25:05.954939 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:05.954906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" event={"ID":"d6844d54-bda4-46e7-8add-6f072c892ed1","Type":"ContainerStarted","Data":"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"}
Sep 09 21:25:05.955332 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:05.954974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" event={"ID":"d6844d54-bda4-46e7-8add-6f072c892ed1","Type":"ContainerStarted","Data":"5ac38a12398cab2a7dbff55c012af1c3f4dd9120d6f4df34ba442dc541fd8bde"}
Sep 09 21:25:05.983609 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:05.983561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" podStartSLOduration=1.983546824 podStartE2EDuration="1.983546824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:25:05.981991724 +0000 UTC m=+1922.733536645" watchObservedRunningTime="2026-09-09 21:25:05.983546824 +0000 UTC m=+1922.735091745"
Sep 09 21:25:06.906606 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:06.906569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:14.906875 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:14.906839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:14.919641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:14.919613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:15.006268 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:15.006238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:25:26.346651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.346580    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:25:26.350468 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.350440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.353696 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.353674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 21:25:26.353836 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.353713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-dqp7v\""
Sep 09 21:25:26.354921 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.354899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 21:25:26.355142 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.355122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 21:25:26.355213 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.355184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-dndtl\""
Sep 09 21:25:26.361459 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.361439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:25:26.429400 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.429372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkpt\" (UniqueName: \"kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.429594 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.429431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.530755 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.530718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.530908 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.530807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkpt\" (UniqueName: \"kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.533197 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.533161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.540981 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.540956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkpt\" (UniqueName: \"kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.662796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.662761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:26.934890 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:26.934858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:25:26.936985 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:25:26.936942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce602000_fb36_4ecb_b4d1_5f4cb9a810d5.slice/crio-23c47fd99aa536384350f10d5bc20820bb9c61ad8027d2840ebbae4274a85ee7 WatchSource:0}: Error finding container 23c47fd99aa536384350f10d5bc20820bb9c61ad8027d2840ebbae4274a85ee7: Status 404 returned error can't find the container with id 23c47fd99aa536384350f10d5bc20820bb9c61ad8027d2840ebbae4274a85ee7
Sep 09 21:25:27.053013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:27.052975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" event={"ID":"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5","Type":"ContainerStarted","Data":"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"}
Sep 09 21:25:27.053190 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:27.053040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" event={"ID":"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5","Type":"ContainerStarted","Data":"23c47fd99aa536384350f10d5bc20820bb9c61ad8027d2840ebbae4274a85ee7"}
Sep 09 21:25:27.053190 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:27.053068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:25:27.079930 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:27.079882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podStartSLOduration=1.07986469 podStartE2EDuration="1.07986469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:25:27.078040892 +0000 UTC m=+1943.829585784" watchObservedRunningTime="2026-09-09 21:25:27.07986469 +0000 UTC m=+1943.831409639"
Sep 09 21:25:55.905094 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:55.905039    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:25:58.059019 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:25:58.058984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:26:10.924760 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:10.924695    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerName="main" containerID="cri-o://04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f" gracePeriod=45
Sep 09 21:26:10.995428 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:10.995392    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 09 21:26:11.171465 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.171441    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:26:11.252176 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.252095    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerID="04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f" exitCode=0
Sep 09 21:26:11.252176 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.252141    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"
Sep 09 21:26:11.252176 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.252160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" event={"ID":"d6844d54-bda4-46e7-8add-6f072c892ed1","Type":"ContainerDied","Data":"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"}
Sep 09 21:26:11.252395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.252200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr" event={"ID":"d6844d54-bda4-46e7-8add-6f072c892ed1","Type":"ContainerDied","Data":"5ac38a12398cab2a7dbff55c012af1c3f4dd9120d6f4df34ba442dc541fd8bde"}
Sep 09 21:26:11.252395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.252223    2563 scope.go:117] "RemoveContainer" containerID="04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"
Sep 09 21:26:11.260494 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.260467    2563 scope.go:117] "RemoveContainer" containerID="04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"
Sep 09 21:26:11.260749 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:26:11.260726    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f\": container with ID starting with 04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f not found: ID does not exist" containerID="04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"
Sep 09 21:26:11.260794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.260757    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f"} err="failed to get container status \"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f\": rpc error: code = NotFound desc = could not find container \"04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f\": container with ID starting with 04eccc6e5b7598e14e82a012ea7522f382fcdd41cecb58c302630bcf9354e49f not found: ID does not exist"
Sep 09 21:26:11.297651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.297754 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.297754 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.297754 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.297754 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.297953 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77kvz\" (UniqueName: \"kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz\") pod \"d6844d54-bda4-46e7-8add-6f072c892ed1\" (UID: \"d6844d54-bda4-46e7-8add-6f072c892ed1\") "
Sep 09 21:26:11.298007 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.297975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:11.298080 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.298059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:11.298130 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.298076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home" (OuterVolumeSpecName: "home") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:11.299893 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.299865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm" (OuterVolumeSpecName: "dshm") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:11.299976 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.299919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz" (OuterVolumeSpecName: "kube-api-access-77kvz") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "kube-api-access-77kvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:26:11.299976 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.299937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6844d54-bda4-46e7-8add-6f072c892ed1" (UID: "d6844d54-bda4-46e7-8add-6f072c892ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:11.398871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398827    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.398871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398866    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kvz\" (UniqueName: \"kubernetes.io/projected/d6844d54-bda4-46e7-8add-6f072c892ed1-kube-api-access-77kvz\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.398871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398877    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.399076 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398906    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.399076 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398915    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6844d54-bda4-46e7-8add-6f072c892ed1-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.399076 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.398922    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6844d54-bda4-46e7-8add-6f072c892ed1-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:26:11.579020 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.578991    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:26:11.584263 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.584239    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-5bntr"]
Sep 09 21:26:11.717736 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:26:11.717700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" path="/var/lib/kubelet/pods/d6844d54-bda4-46e7-8add-6f072c892ed1/volumes"
Sep 09 21:27:05.478387 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.478343    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:27:05.478965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.478942    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerName="main"
Sep 09 21:27:05.479034 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.478968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerName="main"
Sep 09 21:27:05.479088 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.479046    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6844d54-bda4-46e7-8add-6f072c892ed1" containerName="main"
Sep 09 21:27:05.483275 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.483251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.488370 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.488346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-kwhkw\""
Sep 09 21:27:05.488370 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.488362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 09 21:27:05.488588 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.488369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 09 21:27:05.488750 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.488734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 09 21:27:05.499611 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.499592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:27:05.637760 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.637760 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.638047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.638047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.638047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.638047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.637898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjlwc\" (UniqueName: \"kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738677 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738677 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738677 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjlwc\" (UniqueName: \"kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.738965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.738741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.739155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.739098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.739155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.739131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.739155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.739140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.740967 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.740940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.741133 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.741114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.747843 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.747820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjlwc\" (UniqueName: \"kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc\") pod \"decom-v2-kserve-77f867d787-tpldh\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:05.793643 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:05.793612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:06.067687 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:06.067661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:27:06.069551 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:27:06.069523    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd33da2_e996_4330_bda4_c5a32648804b.slice/crio-24eedd74ffd6aae9044405c893d9173d46f080d2a85ad7631770052e294c477e WatchSource:0}: Error finding container 24eedd74ffd6aae9044405c893d9173d46f080d2a85ad7631770052e294c477e: Status 404 returned error can't find the container with id 24eedd74ffd6aae9044405c893d9173d46f080d2a85ad7631770052e294c477e
Sep 09 21:27:06.513530 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:06.513462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" event={"ID":"7dd33da2-e996-4330-bda4-c5a32648804b","Type":"ContainerStarted","Data":"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"}
Sep 09 21:27:06.514054 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:06.513591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" event={"ID":"7dd33da2-e996-4330-bda4-c5a32648804b","Type":"ContainerStarted","Data":"24eedd74ffd6aae9044405c893d9173d46f080d2a85ad7631770052e294c477e"}
Sep 09 21:27:06.537272 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:06.537218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" podStartSLOduration=1.537202248 podStartE2EDuration="1.537202248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:06.535034246 +0000 UTC m=+2043.286579155" watchObservedRunningTime="2026-09-09 21:27:06.537202248 +0000 UTC m=+2043.288747157"
Sep 09 21:27:06.794130 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:06.794042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:15.794394 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:15.794355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:15.806354 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:15.806331    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:16.567323 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:16.567292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:27:40.291594 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:40.291519    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:27:48.060192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:48.060144    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:27:54.733506 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:54.733451    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:27:55.317644 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:55.317607    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main" containerID="cri-o://9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35" gracePeriod=45
Sep 09 21:27:58.059551 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:27:58.059512    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:28:03.785558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:03.785532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:28:03.790352 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:03.790330    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:28:08.059480 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:08.059439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:28:08.059889 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:08.059568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:28:09.753439 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:09.753401    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" podUID="7dd33da2-e996-4330-bda4-c5a32648804b" containerName="main" containerID="cri-o://cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15" gracePeriod=45
Sep 09 21:28:09.994299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:09.994269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:28:10.072044 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.071954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072044 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjlwc\" (UniqueName: \"kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home\") pod \"7dd33da2-e996-4330-bda4-c5a32648804b\" (UID: \"7dd33da2-e996-4330-bda4-c5a32648804b\") "
Sep 09 21:28:10.072519 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:10.072519 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:10.072519 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.072474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home" (OuterVolumeSpecName: "home") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:10.074334 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.074310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:28:10.074426 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.074363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc" (OuterVolumeSpecName: "kube-api-access-gjlwc") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "kube-api-access-gjlwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:28:10.074426 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.074381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm" (OuterVolumeSpecName: "dshm") pod "7dd33da2-e996-4330-bda4-c5a32648804b" (UID: "7dd33da2-e996-4330-bda4-c5a32648804b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:28:10.173046 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173009    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dd33da2-e996-4330-bda4-c5a32648804b-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.173046 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173039    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.173046 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173049    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjlwc\" (UniqueName: \"kubernetes.io/projected/7dd33da2-e996-4330-bda4-c5a32648804b-kube-api-access-gjlwc\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.173290 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173058    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.173290 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173067    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.173290 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.173074    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dd33da2-e996-4330-bda4-c5a32648804b-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:10.806871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.806822    2563 generic.go:356] "Generic (PLEG): container finished" podID="7dd33da2-e996-4330-bda4-c5a32648804b" containerID="cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15" exitCode=0
Sep 09 21:28:10.806871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.806870    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"
Sep 09 21:28:10.807377 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.806896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" event={"ID":"7dd33da2-e996-4330-bda4-c5a32648804b","Type":"ContainerDied","Data":"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"}
Sep 09 21:28:10.807377 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.806969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh" event={"ID":"7dd33da2-e996-4330-bda4-c5a32648804b","Type":"ContainerDied","Data":"24eedd74ffd6aae9044405c893d9173d46f080d2a85ad7631770052e294c477e"}
Sep 09 21:28:10.807377 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.807001    2563 scope.go:117] "RemoveContainer" containerID="cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"
Sep 09 21:28:10.815753 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.815724    2563 scope.go:117] "RemoveContainer" containerID="cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"
Sep 09 21:28:10.815984 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:28:10.815963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15\": container with ID starting with cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15 not found: ID does not exist" containerID="cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"
Sep 09 21:28:10.816045 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.815996    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15"} err="failed to get container status \"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15\": rpc error: code = NotFound desc = could not find container \"cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15\": container with ID starting with cea5a55713ec7d3240e16ccb46cca44b4954af6667247a298720f5e1ff76ad15 not found: ID does not exist"
Sep 09 21:28:10.830157 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.830133    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:28:10.834878 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:10.834856    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-tpldh"]
Sep 09 21:28:11.716297 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:11.716262    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd33da2-e996-4330-bda4-c5a32648804b" path="/var/lib/kubelet/pods/7dd33da2-e996-4330-bda4-c5a32648804b/volumes"
Sep 09 21:28:18.059954 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:18.059908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:28:25.570074 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.570050    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:28:25.701475 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.701441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrkpt\" (UniqueName: \"kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt\") pod \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") "
Sep 09 21:28:25.701666 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.701504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs\") pod \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\" (UID: \"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5\") "
Sep 09 21:28:25.703669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.703635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt" (OuterVolumeSpecName: "kube-api-access-lrkpt") pod "ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" (UID: "ce602000-fb36-4ecb-b4d1-5f4cb9a810d5"). InnerVolumeSpecName "kube-api-access-lrkpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:28:25.703791 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.703713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" (UID: "ce602000-fb36-4ecb-b4d1-5f4cb9a810d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:28:25.802207 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.802172    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrkpt\" (UniqueName: \"kubernetes.io/projected/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-kube-api-access-lrkpt\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:25.802207 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.802199    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:28:25.882977 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.882946    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerID="9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35" exitCode=0
Sep 09 21:28:25.883142 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.883000    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"
Sep 09 21:28:25.883142 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.883038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" event={"ID":"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5","Type":"ContainerDied","Data":"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"}
Sep 09 21:28:25.883142 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.883129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm" event={"ID":"ce602000-fb36-4ecb-b4d1-5f4cb9a810d5","Type":"ContainerDied","Data":"23c47fd99aa536384350f10d5bc20820bb9c61ad8027d2840ebbae4274a85ee7"}
Sep 09 21:28:25.883268 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.883157    2563 scope.go:117] "RemoveContainer" containerID="9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"
Sep 09 21:28:25.891848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.891830    2563 scope.go:117] "RemoveContainer" containerID="9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"
Sep 09 21:28:25.892126 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:28:25.892105    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35\": container with ID starting with 9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35 not found: ID does not exist" containerID="9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"
Sep 09 21:28:25.892192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.892137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35"} err="failed to get container status \"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35\": rpc error: code = NotFound desc = could not find container \"9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35\": container with ID starting with 9a4fe34d7a9334b93ded8077aa926a67edbf07760b698aa3f055c64baf463d35 not found: ID does not exist"
Sep 09 21:28:25.902510 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.902473    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:28:25.908340 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:25.908316    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b698jqnm"]
Sep 09 21:28:26.216909 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.216872    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:28:26.217261 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217243    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd33da2-e996-4330-bda4-c5a32648804b" containerName="main"
Sep 09 21:28:26.217348 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217264    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd33da2-e996-4330-bda4-c5a32648804b" containerName="main"
Sep 09 21:28:26.217348 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217276    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main"
Sep 09 21:28:26.217348 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217285    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main"
Sep 09 21:28:26.217527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217398    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd33da2-e996-4330-bda4-c5a32648804b" containerName="main"
Sep 09 21:28:26.217527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.217411    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" containerName="main"
Sep 09 21:28:26.221588 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.221566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.227082 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.227058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-8nb68\""
Sep 09 21:28:26.227189 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.227171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 09 21:28:26.228305 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.228284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 09 21:28:26.228525 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.228304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 09 21:28:26.251036 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.251008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:28:26.407595 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.407595 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.407822 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.407822 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.407822 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7g2h\" (UniqueName: \"kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.407822 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.407799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508271 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508271 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508271 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7g2h\" (UniqueName: \"kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508803 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.508853 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.508806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.510587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.510568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.510756 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.510738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.518077 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.518045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7g2h\" (UniqueName: \"kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h\") pod \"leave-v1-kserve-65684d4f5-72h5q\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.531834 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.531809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:26.821368 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.821339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:28:26.823850 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:28:26.823826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1ba9f6_e425_4c84_b469_4177611365d8.slice/crio-0ec824e8702d86bd9f7d76915274e5581457f9987ebc25b0ce8742b30115780d WatchSource:0}: Error finding container 0ec824e8702d86bd9f7d76915274e5581457f9987ebc25b0ce8742b30115780d: Status 404 returned error can't find the container with id 0ec824e8702d86bd9f7d76915274e5581457f9987ebc25b0ce8742b30115780d
Sep 09 21:28:26.890034 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.890000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" event={"ID":"6e1ba9f6-e425-4c84-b469-4177611365d8","Type":"ContainerStarted","Data":"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"}
Sep 09 21:28:26.890152 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.890055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" event={"ID":"6e1ba9f6-e425-4c84-b469-4177611365d8","Type":"ContainerStarted","Data":"0ec824e8702d86bd9f7d76915274e5581457f9987ebc25b0ce8742b30115780d"}
Sep 09 21:28:26.917059 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:26.917016    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" podStartSLOduration=0.91700004 podStartE2EDuration="917.00004ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:26.91557747 +0000 UTC m=+2123.667122382" watchObservedRunningTime="2026-09-09 21:28:26.91700004 +0000 UTC m=+2123.668544948"
Sep 09 21:28:27.532517 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:27.532459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:27.714898 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:27.714866    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce602000-fb36-4ecb-b4d1-5f4cb9a810d5" path="/var/lib/kubelet/pods/ce602000-fb36-4ecb-b4d1-5f4cb9a810d5/volumes"
Sep 09 21:28:29.199973 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.199935    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:28:29.203273 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.203256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.207956 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.207934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 09 21:28:29.223689 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.223664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:28:29.229670 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.229780 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f229d\" (UniqueName: \"kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.229780 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.229780 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.229909 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.229909 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.229842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330292 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330292 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f229d\" (UniqueName: \"kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330558 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330771 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330846 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.330906 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.330833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.332748 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.332724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.332859 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.332784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.339439 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.339415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f229d\" (UniqueName: \"kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d\") pod \"leave-v2-kserve-556b4dddf9-4cjcq\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.513811 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.513712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:29.792335 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.792262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:28:29.795351 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:28:29.795320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4652f45b_0352_43b7_80bb_eb0c4458079e.slice/crio-0d3349c1ed4391f9351fdba71c2748ee9b5c97ac18ed6ab0dc3739f1f85beaf7 WatchSource:0}: Error finding container 0d3349c1ed4391f9351fdba71c2748ee9b5c97ac18ed6ab0dc3739f1f85beaf7: Status 404 returned error can't find the container with id 0d3349c1ed4391f9351fdba71c2748ee9b5c97ac18ed6ab0dc3739f1f85beaf7
Sep 09 21:28:29.905362 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.905327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" event={"ID":"4652f45b-0352-43b7-80bb-eb0c4458079e","Type":"ContainerStarted","Data":"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"}
Sep 09 21:28:29.905528 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:29.905416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" event={"ID":"4652f45b-0352-43b7-80bb-eb0c4458079e","Type":"ContainerStarted","Data":"0d3349c1ed4391f9351fdba71c2748ee9b5c97ac18ed6ab0dc3739f1f85beaf7"}
Sep 09 21:28:30.514776 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:30.514733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:36.532334 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:36.532292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:36.544809 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:36.544779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:36.572112 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:36.572063    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" podStartSLOduration=7.572048338 podStartE2EDuration="7.572048338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:29.929506965 +0000 UTC m=+2126.681051872" watchObservedRunningTime="2026-09-09 21:28:36.572048338 +0000 UTC m=+2133.323593293"
Sep 09 21:28:36.948721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:36.948692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:28:39.514029 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:39.513987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:39.526612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:39.526583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:39.962937 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:39.962908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:28:53.966937 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.966908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:28:53.970359 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.970336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:53.975223 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.975200    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 21:28:53.975540 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.975518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-zkt28\""
Sep 09 21:28:53.975660 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.975594    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 21:28:53.975660 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.975594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 21:28:53.975798 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.975710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-4lf5w\""
Sep 09 21:28:53.982921 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:53.982900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:28:54.031050 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.031017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.031203 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.031090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjp67\" (UniqueName: \"kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.131560 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.131529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.131708 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.131610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjp67\" (UniqueName: \"kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.134025 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.133987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.149784 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.149756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjp67\" (UniqueName: \"kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.281164 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.281071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:54.553462 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:54.553437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:28:54.555581 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:28:54.555553    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dd0965_b2e3_4963_a28f_d0e2d1c1075b.slice/crio-815e76c1a1f3486151b47e9698d9b3e6646564766bc82900aaf87064937b845a WatchSource:0}: Error finding container 815e76c1a1f3486151b47e9698d9b3e6646564766bc82900aaf87064937b845a: Status 404 returned error can't find the container with id 815e76c1a1f3486151b47e9698d9b3e6646564766bc82900aaf87064937b845a
Sep 09 21:28:55.023841 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:55.023809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" event={"ID":"17dd0965-b2e3-4963-a28f-d0e2d1c1075b","Type":"ContainerStarted","Data":"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"}
Sep 09 21:28:55.024231 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:55.023863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" event={"ID":"17dd0965-b2e3-4963-a28f-d0e2d1c1075b","Type":"ContainerStarted","Data":"815e76c1a1f3486151b47e9698d9b3e6646564766bc82900aaf87064937b845a"}
Sep 09 21:28:55.024231 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:55.023907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:28:55.047869 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:28:55.047817    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podStartSLOduration=2.047799353 podStartE2EDuration="2.047799353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:55.047166804 +0000 UTC m=+2151.798711713" watchObservedRunningTime="2026-09-09 21:28:55.047799353 +0000 UTC m=+2151.799344263"
Sep 09 21:29:10.206359 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:10.206317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:29:11.360041 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:11.360004    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:29:25.225807 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.225740    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" podUID="6e1ba9f6-e425-4c84-b469-4177611365d8" containerName="main" containerID="cri-o://445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d" gracePeriod=45
Sep 09 21:29:25.470162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.470135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:29:25.604800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.604800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.604984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.604984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604840    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7g2h\" (UniqueName: \"kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.604984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.604984 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.604919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm\") pod \"6e1ba9f6-e425-4c84-b469-4177611365d8\" (UID: \"6e1ba9f6-e425-4c84-b469-4177611365d8\") "
Sep 09 21:29:25.605140 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home" (OuterVolumeSpecName: "home") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.605140 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.605140 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.605261 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605212    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.605261 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605233    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.605372 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.605265    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.607071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.607047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm" (OuterVolumeSpecName: "dshm") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.607205 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.607083    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:29:25.607205 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.607086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h" (OuterVolumeSpecName: "kube-api-access-j7g2h") pod "6e1ba9f6-e425-4c84-b469-4177611365d8" (UID: "6e1ba9f6-e425-4c84-b469-4177611365d8"). InnerVolumeSpecName "kube-api-access-j7g2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:29:25.706269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.706234    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7g2h\" (UniqueName: \"kubernetes.io/projected/6e1ba9f6-e425-4c84-b469-4177611365d8-kube-api-access-j7g2h\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.706269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.706264    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1ba9f6-e425-4c84-b469-4177611365d8-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.706437 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:25.706280    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1ba9f6-e425-4c84-b469-4177611365d8-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.032095 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.032067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:29:26.169107 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.169074    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e1ba9f6-e425-4c84-b469-4177611365d8" containerID="445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d" exitCode=0
Sep 09 21:29:26.169269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.169116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" event={"ID":"6e1ba9f6-e425-4c84-b469-4177611365d8","Type":"ContainerDied","Data":"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"}
Sep 09 21:29:26.169269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.169153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q" event={"ID":"6e1ba9f6-e425-4c84-b469-4177611365d8","Type":"ContainerDied","Data":"0ec824e8702d86bd9f7d76915274e5581457f9987ebc25b0ce8742b30115780d"}
Sep 09 21:29:26.169269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.169175    2563 scope.go:117] "RemoveContainer" containerID="445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"
Sep 09 21:29:26.169269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.169119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"
Sep 09 21:29:26.179480 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.179438    2563 scope.go:117] "RemoveContainer" containerID="445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"
Sep 09 21:29:26.179896 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:29:26.179873    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d\": container with ID starting with 445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d not found: ID does not exist" containerID="445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"
Sep 09 21:29:26.179993 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.179905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d"} err="failed to get container status \"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d\": rpc error: code = NotFound desc = could not find container \"445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d\": container with ID starting with 445f3aad09a3883b9e46f6041d6fda3c1eaf0932b022acbe126366a7efe9c33d not found: ID does not exist"
Sep 09 21:29:26.191545 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.191524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:29:26.200277 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.200251    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-72h5q"]
Sep 09 21:29:26.379291 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.379259    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" podUID="4652f45b-0352-43b7-80bb-eb0c4458079e" containerName="main" containerID="cri-o://93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c" gracePeriod=45
Sep 09 21:29:26.632953 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.632893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:29:26.815474 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815438    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f229d\" (UniqueName: \"kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d\") pod \"4652f45b-0352-43b7-80bb-eb0c4458079e\" (UID: \"4652f45b-0352-43b7-80bb-eb0c4458079e\") "
Sep 09 21:29:26.815959 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:26.815959 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:26.816073 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.815957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home" (OuterVolumeSpecName: "home") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:26.817716 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.817691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:29:26.817836 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.817755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm" (OuterVolumeSpecName: "dshm") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:26.817883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.817838    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d" (OuterVolumeSpecName: "kube-api-access-f229d") pod "4652f45b-0352-43b7-80bb-eb0c4458079e" (UID: "4652f45b-0352-43b7-80bb-eb0c4458079e"). InnerVolumeSpecName "kube-api-access-f229d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:29:26.916641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916570    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4652f45b-0352-43b7-80bb-eb0c4458079e-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.916641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916598    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.916641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916611    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f229d\" (UniqueName: \"kubernetes.io/projected/4652f45b-0352-43b7-80bb-eb0c4458079e-kube-api-access-f229d\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.916641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916622    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.916641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916636    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:26.916872 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:26.916648    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4652f45b-0352-43b7-80bb-eb0c4458079e-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:29:27.175850 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.175766    2563 generic.go:356] "Generic (PLEG): container finished" podID="4652f45b-0352-43b7-80bb-eb0c4458079e" containerID="93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c" exitCode=0
Sep 09 21:29:27.175850 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.175810    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"
Sep 09 21:29:27.176073 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.175842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" event={"ID":"4652f45b-0352-43b7-80bb-eb0c4458079e","Type":"ContainerDied","Data":"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"}
Sep 09 21:29:27.176073 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.175893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq" event={"ID":"4652f45b-0352-43b7-80bb-eb0c4458079e","Type":"ContainerDied","Data":"0d3349c1ed4391f9351fdba71c2748ee9b5c97ac18ed6ab0dc3739f1f85beaf7"}
Sep 09 21:29:27.176073 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.175921    2563 scope.go:117] "RemoveContainer" containerID="93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"
Sep 09 21:29:27.184526 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.184507    2563 scope.go:117] "RemoveContainer" containerID="93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"
Sep 09 21:29:27.184787 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:29:27.184766    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c\": container with ID starting with 93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c not found: ID does not exist" containerID="93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"
Sep 09 21:29:27.184846 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.184796    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c"} err="failed to get container status \"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c\": rpc error: code = NotFound desc = could not find container \"93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c\": container with ID starting with 93d4abfbaa7a362a9f5f0bea35179a0afb64ab7ca2d8ff7a75451e40a1a3030c not found: ID does not exist"
Sep 09 21:29:27.199288 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.199265    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:29:27.203509 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.203475    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-4cjcq"]
Sep 09 21:29:27.715737 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.715704    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4652f45b-0352-43b7-80bb-eb0c4458079e" path="/var/lib/kubelet/pods/4652f45b-0352-43b7-80bb-eb0c4458079e/volumes"
Sep 09 21:29:27.716219 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:27.716204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1ba9f6-e425-4c84-b469-4177611365d8" path="/var/lib/kubelet/pods/6e1ba9f6-e425-4c84-b469-4177611365d8/volumes"
Sep 09 21:29:36.065855 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.065822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4652f45b-0352-43b7-80bb-eb0c4458079e" containerName="main"
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4652f45b-0352-43b7-80bb-eb0c4458079e" containerName="main"
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066170    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ba9f6-e425-4c84-b469-4177611365d8" containerName="main"
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066176    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ba9f6-e425-4c84-b469-4177611365d8" containerName="main"
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066239    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4652f45b-0352-43b7-80bb-eb0c4458079e" containerName="main"
Sep 09 21:29:36.066258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.066249    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1ba9f6-e425-4c84-b469-4177611365d8" containerName="main"
Sep 09 21:29:36.070645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.070622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.073834 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.073815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-gnlgb\""
Sep 09 21:29:36.074956 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.074935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 09 21:29:36.075074 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.075053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 09 21:29:36.075146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.075120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 09 21:29:36.081298 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.081280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:29:36.084317 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.084415 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.084415 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.084415 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vn2j\" (UniqueName: \"kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.084555 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.084555 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.084494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185161 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vn2j\" (UniqueName: \"kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185620 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185711 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.185776 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.185718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.187512 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.187476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.187763 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.187745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.194108 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.194085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vn2j\" (UniqueName: \"kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j\") pod \"tri-v2-kserve-dcb554bf5-9jrdf\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.381145 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.381105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:36.657615 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:36.657590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:29:36.659280 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:29:36.659253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef59f6c_ae7b_4bb2_ae0f_d289acba5b13.slice/crio-c7562971254f9f031d646dcecd1f1aefde34ab9ca0331aa06f75831fe1bd4f3b WatchSource:0}: Error finding container c7562971254f9f031d646dcecd1f1aefde34ab9ca0331aa06f75831fe1bd4f3b: Status 404 returned error can't find the container with id c7562971254f9f031d646dcecd1f1aefde34ab9ca0331aa06f75831fe1bd4f3b
Sep 09 21:29:37.227400 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:37.227360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" event={"ID":"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13","Type":"ContainerStarted","Data":"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"}
Sep 09 21:29:37.227892 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:37.227430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" event={"ID":"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13","Type":"ContainerStarted","Data":"c7562971254f9f031d646dcecd1f1aefde34ab9ca0331aa06f75831fe1bd4f3b"}
Sep 09 21:29:37.251809 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:37.251758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" podStartSLOduration=1.251737988 podStartE2EDuration="1.251737988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:37.251054086 +0000 UTC m=+2194.002598995" watchObservedRunningTime="2026-09-09 21:29:37.251737988 +0000 UTC m=+2194.003282947"
Sep 09 21:29:37.381421 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:37.381385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:46.382136 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:46.382100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:46.394656 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:46.394632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:29:47.282059 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:29:47.282022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:30:02.494513 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.494458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:30:02.498072 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.498048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.501082 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.501050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 09 21:30:02.513392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.513359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:30:02.599265 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gx57\" (UniqueName: \"kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.599445 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.599445 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.599445 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.599626 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.599626 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.599532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700517 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gx57\" (UniqueName: \"kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.700725 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.701002 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.701067 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.700990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.701067 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.701038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.702824 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.702801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.703020 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.703000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.711069 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.711047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gx57\" (UniqueName: \"kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57\") pod \"tri-v3-kserve-798c88c66c-ttwwk\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:02.808551 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:02.808408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:03.087122 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:03.087096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:30:03.089207 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:30:03.089176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b1632d_62b8_4e63_803e_2c76c4ec0194.slice/crio-b9b3ef645b61e1a729fdb4c4ed7daee2e5022d41b618537c6a81ceae099e1b16 WatchSource:0}: Error finding container b9b3ef645b61e1a729fdb4c4ed7daee2e5022d41b618537c6a81ceae099e1b16: Status 404 returned error can't find the container with id b9b3ef645b61e1a729fdb4c4ed7daee2e5022d41b618537c6a81ceae099e1b16
Sep 09 21:30:03.344352 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:03.344266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" event={"ID":"a8b1632d-62b8-4e63-803e-2c76c4ec0194","Type":"ContainerStarted","Data":"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"}
Sep 09 21:30:03.344352 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:03.344315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" event={"ID":"a8b1632d-62b8-4e63-803e-2c76c4ec0194","Type":"ContainerStarted","Data":"b9b3ef645b61e1a729fdb4c4ed7daee2e5022d41b618537c6a81ceae099e1b16"}
Sep 09 21:30:03.370149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:03.370102    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" podStartSLOduration=1.370087498 podStartE2EDuration="1.370087498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:30:03.368726734 +0000 UTC m=+2220.120271645" watchObservedRunningTime="2026-09-09 21:30:03.370087498 +0000 UTC m=+2220.121632406"
Sep 09 21:30:03.809021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:03.808988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:12.809064 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:12.809026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:12.821612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:12.821587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:13.398391 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:13.398355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:30:22.824861 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:22.824826    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:30:26.031836 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:26.031748    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:30:36.031991 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:36.031949    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:30:37.845358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:37.845318    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" containerID="cri-o://f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60" gracePeriod=45
Sep 09 21:30:46.032106 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:46.032068    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:30:46.032561 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:46.032174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:30:56.031952 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:30:56.031910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:31:02.789274 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:02.789240    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:31:04.080445 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:04.080412    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:31:06.032230 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:06.032189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:31:08.092693 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.092665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:31:08.254961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.254923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs\") pod \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") "
Sep 09 21:31:08.255150 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.255011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjp67\" (UniqueName: \"kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67\") pod \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\" (UID: \"17dd0965-b2e3-4963-a28f-d0e2d1c1075b\") "
Sep 09 21:31:08.257134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.257104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67" (OuterVolumeSpecName: "kube-api-access-rjp67") pod "17dd0965-b2e3-4963-a28f-d0e2d1c1075b" (UID: "17dd0965-b2e3-4963-a28f-d0e2d1c1075b"). InnerVolumeSpecName "kube-api-access-rjp67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:08.257134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.257107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17dd0965-b2e3-4963-a28f-d0e2d1c1075b" (UID: "17dd0965-b2e3-4963-a28f-d0e2d1c1075b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:08.356272 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.356231    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjp67\" (UniqueName: \"kubernetes.io/projected/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-kube-api-access-rjp67\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:08.356272 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.356264    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dd0965-b2e3-4963-a28f-d0e2d1c1075b-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:08.642356 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.642315    2563 generic.go:356] "Generic (PLEG): container finished" podID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerID="f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60" exitCode=0
Sep 09 21:31:08.642587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.642363    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"
Sep 09 21:31:08.642587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.642389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" event={"ID":"17dd0965-b2e3-4963-a28f-d0e2d1c1075b","Type":"ContainerDied","Data":"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"}
Sep 09 21:31:08.642587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.642451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p" event={"ID":"17dd0965-b2e3-4963-a28f-d0e2d1c1075b","Type":"ContainerDied","Data":"815e76c1a1f3486151b47e9698d9b3e6646564766bc82900aaf87064937b845a"}
Sep 09 21:31:08.642587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.642481    2563 scope.go:117] "RemoveContainer" containerID="f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"
Sep 09 21:31:08.651742 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.651724    2563 scope.go:117] "RemoveContainer" containerID="f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"
Sep 09 21:31:08.651978 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:31:08.651957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60\": container with ID starting with f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60 not found: ID does not exist" containerID="f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"
Sep 09 21:31:08.652028 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.651987    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60"} err="failed to get container status \"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60\": rpc error: code = NotFound desc = could not find container \"f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60\": container with ID starting with f54499be7793fb692dd67abc7ab5db72c4d8b82b71b41c01563d8d1ac8dc1f60 not found: ID does not exist"
Sep 09 21:31:08.666398 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.666374    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:31:08.670730 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:08.670710    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785qm56p"]
Sep 09 21:31:09.715903 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:09.715861    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" path="/var/lib/kubelet/pods/17dd0965-b2e3-4963-a28f-d0e2d1c1075b/volumes"
Sep 09 21:31:12.496259 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.496221    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:31:12.496695 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.496676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main"
Sep 09 21:31:12.496742 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.496697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main"
Sep 09 21:31:12.496775 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.496749    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd0965-b2e3-4963-a28f-d0e2d1c1075b" containerName="main"
Sep 09 21:31:12.499706 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.499687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.502666 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.502638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-rs9sr\""
Sep 09 21:31:12.503831 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.503812    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 09 21:31:12.503831 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.503837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 09 21:31:12.503991 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.503859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 09 21:31:12.516192 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.516166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:31:12.592421 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.592619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.592619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.592619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.592619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.592619 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.592605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8mnf\" (UniqueName: \"kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693948 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693948 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693948 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8mnf\" (UniqueName: \"kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.693948 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.693866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.694177 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.694077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.694177 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.694139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.694279 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.694206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.696106 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.696084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.696419 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.696403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.708536 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.708516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8mnf\" (UniqueName: \"kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf\") pod \"late-v1-kserve-56cbc8456c-qhqh2\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:12.810517 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:12.810408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:13.085579 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:13.085553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:31:13.087619 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:31:13.087591    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca46f85b_cf8b_4383_a30a_10de382d1f67.slice/crio-62c420dbc2ce668f7b3f5bf9f3b452c37f2e723c877ddbc19953ccb955cb5515 WatchSource:0}: Error finding container 62c420dbc2ce668f7b3f5bf9f3b452c37f2e723c877ddbc19953ccb955cb5515: Status 404 returned error can't find the container with id 62c420dbc2ce668f7b3f5bf9f3b452c37f2e723c877ddbc19953ccb955cb5515
Sep 09 21:31:13.668981 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:13.668949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" event={"ID":"ca46f85b-cf8b-4383-a30a-10de382d1f67","Type":"ContainerStarted","Data":"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"}
Sep 09 21:31:13.669404 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:13.669000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" event={"ID":"ca46f85b-cf8b-4383-a30a-10de382d1f67","Type":"ContainerStarted","Data":"62c420dbc2ce668f7b3f5bf9f3b452c37f2e723c877ddbc19953ccb955cb5515"}
Sep 09 21:31:13.696896 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:13.696839    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" podStartSLOduration=1.696824591 podStartE2EDuration="1.696824591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:13.695278298 +0000 UTC m=+2290.446823210" watchObservedRunningTime="2026-09-09 21:31:13.696824591 +0000 UTC m=+2290.448369499"
Sep 09 21:31:13.810631 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:13.810599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:17.808611 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:17.808575    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" podUID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" containerName="main" containerID="cri-o://223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998" gracePeriod=45
Sep 09 21:31:18.059430 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.059376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:31:18.244618 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gx57\" (UniqueName: \"kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.244797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.244797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.244797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.244797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.244797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.244774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm\") pod \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\" (UID: \"a8b1632d-62b8-4e63-803e-2c76c4ec0194\") "
Sep 09 21:31:18.245512 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.245398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.245512 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.245465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.245694 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.245541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home" (OuterVolumeSpecName: "home") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.253190 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.252995    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57" (OuterVolumeSpecName: "kube-api-access-4gx57") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "kube-api-access-4gx57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:18.253461 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.253431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm" (OuterVolumeSpecName: "dshm") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:18.253730 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.253707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8b1632d-62b8-4e63-803e-2c76c4ec0194" (UID: "a8b1632d-62b8-4e63-803e-2c76c4ec0194"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.345936    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gx57\" (UniqueName: \"kubernetes.io/projected/a8b1632d-62b8-4e63-803e-2c76c4ec0194-kube-api-access-4gx57\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.345967    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.345977    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.345985    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.345993    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.346021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.346001    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b1632d-62b8-4e63-803e-2c76c4ec0194-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:18.693220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.693183    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" containerID="223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998" exitCode=0
Sep 09 21:31:18.693392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.693241    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"
Sep 09 21:31:18.693392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.693272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" event={"ID":"a8b1632d-62b8-4e63-803e-2c76c4ec0194","Type":"ContainerDied","Data":"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"}
Sep 09 21:31:18.693392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.693343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk" event={"ID":"a8b1632d-62b8-4e63-803e-2c76c4ec0194","Type":"ContainerDied","Data":"b9b3ef645b61e1a729fdb4c4ed7daee2e5022d41b618537c6a81ceae099e1b16"}
Sep 09 21:31:18.693392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.693366    2563 scope.go:117] "RemoveContainer" containerID="223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"
Sep 09 21:31:18.703730 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.703655    2563 scope.go:117] "RemoveContainer" containerID="223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"
Sep 09 21:31:18.704245 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:31:18.704222    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998\": container with ID starting with 223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998 not found: ID does not exist" containerID="223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"
Sep 09 21:31:18.704328 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.704250    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998"} err="failed to get container status \"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998\": rpc error: code = NotFound desc = could not find container \"223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998\": container with ID starting with 223cb0dca5d86f54394e3b4ead5eb1408a59bc9aa5cfa30eb286432684588998 not found: ID does not exist"
Sep 09 21:31:18.722119 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.722089    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:31:18.723650 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:18.723629    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-ttwwk"]
Sep 09 21:31:19.100233 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.100116    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerName="main" containerID="cri-o://af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35" gracePeriod=45
Sep 09 21:31:19.350684 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.350628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:31:19.353630 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353715 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353715 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vn2j\" (UniqueName: \"kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir\") pod \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\" (UID: \"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13\") "
Sep 09 21:31:19.353952 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home" (OuterVolumeSpecName: "home") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:19.354010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.353982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:19.354097 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.354078    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.354149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.354102    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.354149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.354103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:19.355741 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.355717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j" (OuterVolumeSpecName: "kube-api-access-5vn2j") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "kube-api-access-5vn2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:19.355900 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.355883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:19.355954 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.355898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm" (OuterVolumeSpecName: "dshm") pod "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" (UID: "2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:19.454645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.454611    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vn2j\" (UniqueName: \"kubernetes.io/projected/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-kube-api-access-5vn2j\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.454645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.454637    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.454645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.454647    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.454873 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.454656    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:31:19.700736 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.700700    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerID="af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35" exitCode=0
Sep 09 21:31:19.700905 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.700745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" event={"ID":"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13","Type":"ContainerDied","Data":"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"}
Sep 09 21:31:19.700905 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.700751    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"
Sep 09 21:31:19.700905 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.700786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" event={"ID":"2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13","Type":"ContainerDied","Data":"c7562971254f9f031d646dcecd1f1aefde34ab9ca0331aa06f75831fe1bd4f3b"}
Sep 09 21:31:19.700905 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.700806    2563 scope.go:117] "RemoveContainer" containerID="af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"
Sep 09 21:31:19.709410 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.709393    2563 scope.go:117] "RemoveContainer" containerID="af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"
Sep 09 21:31:19.709705 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:31:19.709686    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35\": container with ID starting with af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35 not found: ID does not exist" containerID="af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"
Sep 09 21:31:19.709764 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.709713    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35"} err="failed to get container status \"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35\": rpc error: code = NotFound desc = could not find container \"af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35\": container with ID starting with af8e862e9b82d29ce822dfad7426d91423d93e5c19d1713b21a283ca8dc92c35 not found: ID does not exist"
Sep 09 21:31:19.715604 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.715578    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" path="/var/lib/kubelet/pods/a8b1632d-62b8-4e63-803e-2c76c4ec0194/volumes"
Sep 09 21:31:19.725587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.725567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:31:19.733987 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:19.733969    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf"]
Sep 09 21:31:20.272198 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:20.272155    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9jrdf" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:31:21.715454 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.715417    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" path="/var/lib/kubelet/pods/2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13/volumes"
Sep 09 21:31:21.890259 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890228    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:31:21.890689 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890664    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerName="main"
Sep 09 21:31:21.890689 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890690    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerName="main"
Sep 09 21:31:21.890880 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" containerName="main"
Sep 09 21:31:21.890880 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" containerName="main"
Sep 09 21:31:21.890880 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890833    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef59f6c-ae7b-4bb2-ae0f-d289acba5b13" containerName="main"
Sep 09 21:31:21.890880 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.890842    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b1632d-62b8-4e63-803e-2c76c4ec0194" containerName="main"
Sep 09 21:31:21.895249 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.895228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:21.900376 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.900354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 21:31:21.900376 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.900366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-gfw6x\""
Sep 09 21:31:21.900584 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.900366    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 21:31:21.901186 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.901159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 21:31:21.901688 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.901663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-v9wtf\""
Sep 09 21:31:21.908680 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.908655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:31:21.971964 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.971871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:21.971964 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:21.971920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7s4\" (UniqueName: \"kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.072364 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.072326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7s4\" (UniqueName: \"kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.072542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.072418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.075031 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.075007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.082140 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.082116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7s4\" (UniqueName: \"kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.210407 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.210365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.492759 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.492723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:31:22.494894 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:31:22.494867    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac3380a_ddc5_4759_9718_9778c7a954ee.slice/crio-1b04ebb3ed28a3237f301656c5bea4cbb4dab09f86fe567a5d2bb0a452d43dd6 WatchSource:0}: Error finding container 1b04ebb3ed28a3237f301656c5bea4cbb4dab09f86fe567a5d2bb0a452d43dd6: Status 404 returned error can't find the container with id 1b04ebb3ed28a3237f301656c5bea4cbb4dab09f86fe567a5d2bb0a452d43dd6
Sep 09 21:31:22.718313 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.718276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" event={"ID":"1ac3380a-ddc5-4759-9718-9778c7a954ee","Type":"ContainerStarted","Data":"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"}
Sep 09 21:31:22.718738 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.718327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" event={"ID":"1ac3380a-ddc5-4759-9718-9778c7a954ee","Type":"ContainerStarted","Data":"1b04ebb3ed28a3237f301656c5bea4cbb4dab09f86fe567a5d2bb0a452d43dd6"}
Sep 09 21:31:22.718738 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.718358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:31:22.745639 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.745531    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podStartSLOduration=1.74550992 podStartE2EDuration="1.74550992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:22.744338805 +0000 UTC m=+2299.495883716" watchObservedRunningTime="2026-09-09 21:31:22.74550992 +0000 UTC m=+2299.497054846"
Sep 09 21:31:22.811316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.811260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:22.823567 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:22.823537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:23.735010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:23.734978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:31:53.724617 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:31:53.724583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:32:02.404155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:02.404074    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:32:03.724944 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:03.724907    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:32:13.725395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:13.725356    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:32:17.423290 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:17.423228    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" containerID="cri-o://0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd" gracePeriod=45
Sep 09 21:32:20.588571 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:20.588531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:32:23.725419 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:23.725376    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:32:23.725934 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:23.725505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:32:33.725273 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:33.725227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:32:35.607834 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:35.607770    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerName="main" containerID="cri-o://480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846" gracePeriod=45
Sep 09 21:32:35.724202 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:35.724167    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 09 21:32:35.858839 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:35.858782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:32:36.001585 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001554    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.001744 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.001744 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.001744 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.001744 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.001744 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8mnf\" (UniqueName: \"kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf\") pod \"ca46f85b-cf8b-4383-a30a-10de382d1f67\" (UID: \"ca46f85b-cf8b-4383-a30a-10de382d1f67\") "
Sep 09 21:32:36.002013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:36.002013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:36.002013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.001971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home" (OuterVolumeSpecName: "home") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:36.004039 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.004014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm" (OuterVolumeSpecName: "dshm") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:36.004039 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.004024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf" (OuterVolumeSpecName: "kube-api-access-f8mnf") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "kube-api-access-f8mnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:36.004180 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.004017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca46f85b-cf8b-4383-a30a-10de382d1f67" (UID: "ca46f85b-cf8b-4383-a30a-10de382d1f67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:36.060745 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.060721    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerID="480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846" exitCode=0
Sep 09 21:32:36.060863 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.060774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"
Sep 09 21:32:36.060863 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.060811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" event={"ID":"ca46f85b-cf8b-4383-a30a-10de382d1f67","Type":"ContainerDied","Data":"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"}
Sep 09 21:32:36.060968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.060864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2" event={"ID":"ca46f85b-cf8b-4383-a30a-10de382d1f67","Type":"ContainerDied","Data":"62c420dbc2ce668f7b3f5bf9f3b452c37f2e723c877ddbc19953ccb955cb5515"}
Sep 09 21:32:36.060968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.060886    2563 scope.go:117] "RemoveContainer" containerID="480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"
Sep 09 21:32:36.068953 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.068938    2563 scope.go:117] "RemoveContainer" containerID="480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"
Sep 09 21:32:36.069233 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:32:36.069213    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846\": container with ID starting with 480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846 not found: ID does not exist" containerID="480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"
Sep 09 21:32:36.069278 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.069242    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846"} err="failed to get container status \"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846\": rpc error: code = NotFound desc = could not find container \"480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846\": container with ID starting with 480b53f34f8474a44142a13b19bf5352afb3343ea426a5be9be5f01e8c4c7846 not found: ID does not exist"
Sep 09 21:32:36.096187 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.096166    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:32:36.102790 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102772    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8mnf\" (UniqueName: \"kubernetes.io/projected/ca46f85b-cf8b-4383-a30a-10de382d1f67-kube-api-access-f8mnf\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.102848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102793    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.102848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102804    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.102848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102813    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.102848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102822    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca46f85b-cf8b-4383-a30a-10de382d1f67-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.102848 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.102830    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca46f85b-cf8b-4383-a30a-10de382d1f67-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:36.105439 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:36.105415    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-qhqh2"]
Sep 09 21:32:37.715919 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:37.715879    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" path="/var/lib/kubelet/pods/ca46f85b-cf8b-4383-a30a-10de382d1f67/volumes"
Sep 09 21:32:43.724684 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:43.724646    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:32:47.668895 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.668868    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:32:47.792356 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.792281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq7s4\" (UniqueName: \"kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4\") pod \"1ac3380a-ddc5-4759-9718-9778c7a954ee\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") "
Sep 09 21:32:47.792511 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.792384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs\") pod \"1ac3380a-ddc5-4759-9718-9778c7a954ee\" (UID: \"1ac3380a-ddc5-4759-9718-9778c7a954ee\") "
Sep 09 21:32:47.794476 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.794453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4" (OuterVolumeSpecName: "kube-api-access-cq7s4") pod "1ac3380a-ddc5-4759-9718-9778c7a954ee" (UID: "1ac3380a-ddc5-4759-9718-9778c7a954ee"). InnerVolumeSpecName "kube-api-access-cq7s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:47.794572 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.794557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ac3380a-ddc5-4759-9718-9778c7a954ee" (UID: "1ac3380a-ddc5-4759-9718-9778c7a954ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:47.893091 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.893058    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7s4\" (UniqueName: \"kubernetes.io/projected/1ac3380a-ddc5-4759-9718-9778c7a954ee-kube-api-access-cq7s4\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:47.893091 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:47.893086    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac3380a-ddc5-4759-9718-9778c7a954ee-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:32:48.120340 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.120305    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerID="0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd" exitCode=0
Sep 09 21:32:48.120547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.120343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"
Sep 09 21:32:48.120547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.120358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" event={"ID":"1ac3380a-ddc5-4759-9718-9778c7a954ee","Type":"ContainerDied","Data":"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"}
Sep 09 21:32:48.120547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.120414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2" event={"ID":"1ac3380a-ddc5-4759-9718-9778c7a954ee","Type":"ContainerDied","Data":"1b04ebb3ed28a3237f301656c5bea4cbb4dab09f86fe567a5d2bb0a452d43dd6"}
Sep 09 21:32:48.120547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.120443    2563 scope.go:117] "RemoveContainer" containerID="0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"
Sep 09 21:32:48.128994 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.128977    2563 scope.go:117] "RemoveContainer" containerID="0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"
Sep 09 21:32:48.129257 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:32:48.129240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd\": container with ID starting with 0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd not found: ID does not exist" containerID="0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"
Sep 09 21:32:48.129305 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.129265    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd"} err="failed to get container status \"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd\": rpc error: code = NotFound desc = could not find container \"0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd\": container with ID starting with 0b293686400aa66ebd71360c7ffcd4d07f1940110ebd4f0a635ddce9eea61dfd not found: ID does not exist"
Sep 09 21:32:48.147583 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.147557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:32:48.155811 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:48.155792    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhw8q2"]
Sep 09 21:32:49.716499 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:32:49.716461    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" path="/var/lib/kubelet/pods/1ac3380a-ddc5-4759-9718-9778c7a954ee/volumes"
Sep 09 21:33:03.810612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:03.810580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:33:03.817317 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:03.817293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:33:07.860992 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.860960    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:33:07.861505 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main"
Sep 09 21:33:07.861563 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861510    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main"
Sep 09 21:33:07.861563 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861530    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerName="main"
Sep 09 21:33:07.861563 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861541    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerName="main"
Sep 09 21:33:07.861660 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861641    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ac3380a-ddc5-4759-9718-9778c7a954ee" containerName="main"
Sep 09 21:33:07.861660 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.861652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca46f85b-cf8b-4383-a30a-10de382d1f67" containerName="main"
Sep 09 21:33:07.866763 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.866741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:07.869523 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.869497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 21:33:07.870962 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.870937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 21:33:07.871092 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.870978    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 21:33:07.871092 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.870943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lxc9f\""
Sep 09 21:33:07.871250 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.871229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-wg64v\""
Sep 09 21:33:07.884130 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.884107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:33:07.955559 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.955524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:07.955740 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:07.955584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhc5k\" (UniqueName: \"kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.056924 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.056893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhc5k\" (UniqueName: \"kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.057124 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.056963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.059871 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.059837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.066978 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.066954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhc5k\" (UniqueName: \"kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.178511 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.178470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:08.461637 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:08.461552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:33:08.464477 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:33:08.464446    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f373042_1df4_4051_82e0_dde8d23ce925.slice/crio-5026e3b58cb9810f42bfb8625abb6d6fa2ab40d92d72cabbb15a1ce78f62da57 WatchSource:0}: Error finding container 5026e3b58cb9810f42bfb8625abb6d6fa2ab40d92d72cabbb15a1ce78f62da57: Status 404 returned error can't find the container with id 5026e3b58cb9810f42bfb8625abb6d6fa2ab40d92d72cabbb15a1ce78f62da57
Sep 09 21:33:09.223296 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:09.223260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" event={"ID":"5f373042-1df4-4051-82e0-dde8d23ce925","Type":"ContainerStarted","Data":"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"}
Sep 09 21:33:09.223828 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:09.223320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" event={"ID":"5f373042-1df4-4051-82e0-dde8d23ce925","Type":"ContainerStarted","Data":"5026e3b58cb9810f42bfb8625abb6d6fa2ab40d92d72cabbb15a1ce78f62da57"}
Sep 09 21:33:09.223828 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:09.223359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:09.253389 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:09.253337    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podStartSLOduration=2.253320615 podStartE2EDuration="2.253320615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:09.252241707 +0000 UTC m=+2406.003786615" watchObservedRunningTime="2026-09-09 21:33:09.253320615 +0000 UTC m=+2406.004865524"
Sep 09 21:33:40.230333 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:40.230259    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:33:41.895367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:41.895326    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:33:50.230299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:50.230256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:33:56.916788 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:33:56.916740    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main" containerID="cri-o://1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d" gracePeriod=45
Sep 09 21:34:00.229825 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:00.229786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:34:10.229474 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:10.229439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:34:10.229893 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:10.229568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:34:20.230149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:20.230102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:34:27.163959 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.163932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:34:27.219135 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.219064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhc5k\" (UniqueName: \"kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k\") pod \"5f373042-1df4-4051-82e0-dde8d23ce925\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") "
Sep 09 21:34:27.219135 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.219103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs\") pod \"5f373042-1df4-4051-82e0-dde8d23ce925\" (UID: \"5f373042-1df4-4051-82e0-dde8d23ce925\") "
Sep 09 21:34:27.221232 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.221204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f373042-1df4-4051-82e0-dde8d23ce925" (UID: "5f373042-1df4-4051-82e0-dde8d23ce925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:34:27.221339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.221265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k" (OuterVolumeSpecName: "kube-api-access-rhc5k") pod "5f373042-1df4-4051-82e0-dde8d23ce925" (UID: "5f373042-1df4-4051-82e0-dde8d23ce925"). InnerVolumeSpecName "kube-api-access-rhc5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:34:27.319777 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.319746    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhc5k\" (UniqueName: \"kubernetes.io/projected/5f373042-1df4-4051-82e0-dde8d23ce925-kube-api-access-rhc5k\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:34:27.319777 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.319772    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f373042-1df4-4051-82e0-dde8d23ce925-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:34:27.561902 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.561815    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f373042-1df4-4051-82e0-dde8d23ce925" containerID="1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d" exitCode=0
Sep 09 21:34:27.561902 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.561852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" event={"ID":"5f373042-1df4-4051-82e0-dde8d23ce925","Type":"ContainerDied","Data":"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"}
Sep 09 21:34:27.561902 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.561867    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"
Sep 09 21:34:27.561902 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.561888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr" event={"ID":"5f373042-1df4-4051-82e0-dde8d23ce925","Type":"ContainerDied","Data":"5026e3b58cb9810f42bfb8625abb6d6fa2ab40d92d72cabbb15a1ce78f62da57"}
Sep 09 21:34:27.562236 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.561910    2563 scope.go:117] "RemoveContainer" containerID="1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"
Sep 09 21:34:27.570075 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.570051    2563 scope.go:117] "RemoveContainer" containerID="1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"
Sep 09 21:34:27.570345 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:34:27.570328    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d\": container with ID starting with 1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d not found: ID does not exist" containerID="1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"
Sep 09 21:34:27.570396 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.570353    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d"} err="failed to get container status \"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d\": rpc error: code = NotFound desc = could not find container \"1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d\": container with ID starting with 1c35da7a7ca410d3dff11efb595f3447a063d964fe6c5bb8ffb132d9c9731e0d not found: ID does not exist"
Sep 09 21:34:27.587433 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.587410    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:34:27.589677 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.589655    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheghxfr"]
Sep 09 21:34:27.715369 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:27.715336    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" path="/var/lib/kubelet/pods/5f373042-1df4-4051-82e0-dde8d23ce925/volumes"
Sep 09 21:34:43.022939 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.018183    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:34:43.023535 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.023407    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main"
Sep 09 21:34:43.023599 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.023539    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main"
Sep 09 21:34:43.023642 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.023633    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f373042-1df4-4051-82e0-dde8d23ce925" containerName="main"
Sep 09 21:34:43.026727 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.026707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.029669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.029644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-sznt6\""
Sep 09 21:34:43.030695 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.030678    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 21:34:43.030976 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.030958    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 21:34:43.031065 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.030988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 09 21:34:43.031065 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.030998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-2grph\""
Sep 09 21:34:43.039740 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.039713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:34:43.157787 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.157744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnnx\" (UniqueName: \"kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.157980 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.157818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.258983 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.258942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.259171 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.259090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnnx\" (UniqueName: \"kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.261400 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.261380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.268034 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.268010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnnx\" (UniqueName: \"kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.337245 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.337157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:43.621763 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.621730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:34:43.624815 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:34:43.624787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c2f56f_41b1_4275_a865_9b77c129435f.slice/crio-60bde64eafb8cafd98ebaf603c990b0e1612a4fd9bad1b6c740e607a32e848f6 WatchSource:0}: Error finding container 60bde64eafb8cafd98ebaf603c990b0e1612a4fd9bad1b6c740e607a32e848f6: Status 404 returned error can't find the container with id 60bde64eafb8cafd98ebaf603c990b0e1612a4fd9bad1b6c740e607a32e848f6
Sep 09 21:34:43.636883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:43.636814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" event={"ID":"19c2f56f-41b1-4275-a865-9b77c129435f","Type":"ContainerStarted","Data":"60bde64eafb8cafd98ebaf603c990b0e1612a4fd9bad1b6c740e607a32e848f6"}
Sep 09 21:34:44.642811 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:44.642770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" event={"ID":"19c2f56f-41b1-4275-a865-9b77c129435f","Type":"ContainerStarted","Data":"ececb127c6f8c46fc10e66ec8a8761e608cf92197a290c94bbd8fef73742696a"}
Sep 09 21:34:44.643272 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:44.642896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:34:44.683248 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:34:44.683188    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podStartSLOduration=2.683170761 podStartE2EDuration="2.683170761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:44.681782404 +0000 UTC m=+2501.433327314" watchObservedRunningTime="2026-09-09 21:34:44.683170761 +0000 UTC m=+2501.434715670"
Sep 09 21:35:15.649859 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:15.649827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:35:38.652737 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.652701    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:35:38.656111 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.656091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.660788 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.660761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-lgmhs\""
Sep 09 21:35:38.660913 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.660800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 09 21:35:38.668575 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.668552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:35:38.724527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.724480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.724677 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.724535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4rv\" (UniqueName: \"kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.825925 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.825888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.826110 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.825935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4rv\" (UniqueName: \"kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.828436 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.828407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.840173 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.840137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4rv\" (UniqueName: \"kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:38.967220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:38.967124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:39.256862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:39.256775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:35:39.261004 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:35:39.260977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485a22cb_58be_4151_b30b_0db7b843fca1.slice/crio-e483e091c296c03d94697d36eccb672427bc0befad4fa90a47905ef27fc3b167 WatchSource:0}: Error finding container e483e091c296c03d94697d36eccb672427bc0befad4fa90a47905ef27fc3b167: Status 404 returned error can't find the container with id e483e091c296c03d94697d36eccb672427bc0befad4fa90a47905ef27fc3b167
Sep 09 21:35:39.885977 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:39.885933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" event={"ID":"485a22cb-58be-4151-b30b-0db7b843fca1","Type":"ContainerStarted","Data":"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"}
Sep 09 21:35:39.886408 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:39.885993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" event={"ID":"485a22cb-58be-4151-b30b-0db7b843fca1","Type":"ContainerStarted","Data":"e483e091c296c03d94697d36eccb672427bc0befad4fa90a47905ef27fc3b167"}
Sep 09 21:35:39.886408 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:39.886069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:35:39.912633 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:35:39.912580    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podStartSLOduration=1.912561383 podStartE2EDuration="1.912561383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:39.909249019 +0000 UTC m=+2556.660793939" watchObservedRunningTime="2026-09-09 21:35:39.912561383 +0000 UTC m=+2556.664106292"
Sep 09 21:36:10.893578 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:10.893547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:36:29.691735 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:29.691640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:36:30.892883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:30.892840    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:36:40.893463 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:40.893373    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:36:44.711211 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:44.711154    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" containerID="cri-o://5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a" gracePeriod=45
Sep 09 21:36:50.893534 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:50.893468    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:36:50.893900 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:36:50.893615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:37:00.892793 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:00.892756    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:37:10.893597 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:10.893560    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:37:14.962876 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:14.962848    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:37:15.051016 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.050932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4rv\" (UniqueName: \"kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv\") pod \"485a22cb-58be-4151-b30b-0db7b843fca1\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") "
Sep 09 21:37:15.051016 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.050974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs\") pod \"485a22cb-58be-4151-b30b-0db7b843fca1\" (UID: \"485a22cb-58be-4151-b30b-0db7b843fca1\") "
Sep 09 21:37:15.053125 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.053094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "485a22cb-58be-4151-b30b-0db7b843fca1" (UID: "485a22cb-58be-4151-b30b-0db7b843fca1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:15.053238 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.053211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv" (OuterVolumeSpecName: "kube-api-access-vq4rv") pod "485a22cb-58be-4151-b30b-0db7b843fca1" (UID: "485a22cb-58be-4151-b30b-0db7b843fca1"). InnerVolumeSpecName "kube-api-access-vq4rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:15.152403 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.152371    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4rv\" (UniqueName: \"kubernetes.io/projected/485a22cb-58be-4151-b30b-0db7b843fca1-kube-api-access-vq4rv\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:37:15.152403 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.152397    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485a22cb-58be-4151-b30b-0db7b843fca1-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:37:15.335853 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.335755    2563 generic.go:356] "Generic (PLEG): container finished" podID="485a22cb-58be-4151-b30b-0db7b843fca1" containerID="5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a" exitCode=0
Sep 09 21:37:15.335853 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.335809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"
Sep 09 21:37:15.336081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.335817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" event={"ID":"485a22cb-58be-4151-b30b-0db7b843fca1","Type":"ContainerDied","Data":"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"}
Sep 09 21:37:15.336081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.335950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb" event={"ID":"485a22cb-58be-4151-b30b-0db7b843fca1","Type":"ContainerDied","Data":"e483e091c296c03d94697d36eccb672427bc0befad4fa90a47905ef27fc3b167"}
Sep 09 21:37:15.336081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.335974    2563 scope.go:117] "RemoveContainer" containerID="5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"
Sep 09 21:37:15.345012 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.344991    2563 scope.go:117] "RemoveContainer" containerID="5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"
Sep 09 21:37:15.345256 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:37:15.345238    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a\": container with ID starting with 5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a not found: ID does not exist" containerID="5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"
Sep 09 21:37:15.345305 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.345264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a"} err="failed to get container status \"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a\": rpc error: code = NotFound desc = could not find container \"5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a\": container with ID starting with 5d5039c9b6fa32c7acc81e8fb147689c246e481e8ed733e868253435b4818a1a not found: ID does not exist"
Sep 09 21:37:15.362010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.361984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:37:15.366832 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.366809    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebt5tb"]
Sep 09 21:37:15.715208 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:15.715172    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" path="/var/lib/kubelet/pods/485a22cb-58be-4151-b30b-0db7b843fca1/volumes"
Sep 09 21:37:21.321141 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:21.321103    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:37:25.648866 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:25.648824    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:37:35.649223 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:35.649185    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:37:36.340481 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:36.340441    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" containerID="cri-o://ececb127c6f8c46fc10e66ec8a8761e608cf92197a290c94bbd8fef73742696a" gracePeriod=45
Sep 09 21:37:45.649646 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:45.649605    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:37:45.650069 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:45.649715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:37:55.649092 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:37:55.649055    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:38:03.837299 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:03.837270    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:38:03.844435 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:03.844412    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:38:05.649527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:05.649474    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:38:06.555943 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.555909    2563 generic.go:356] "Generic (PLEG): container finished" podID="19c2f56f-41b1-4275-a865-9b77c129435f" containerID="ececb127c6f8c46fc10e66ec8a8761e608cf92197a290c94bbd8fef73742696a" exitCode=0
Sep 09 21:38:06.556104 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.555988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" event={"ID":"19c2f56f-41b1-4275-a865-9b77c129435f","Type":"ContainerDied","Data":"ececb127c6f8c46fc10e66ec8a8761e608cf92197a290c94bbd8fef73742696a"}
Sep 09 21:38:06.587179 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.587154    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:38:06.689064 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.689025    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfnnx\" (UniqueName: \"kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx\") pod \"19c2f56f-41b1-4275-a865-9b77c129435f\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") "
Sep 09 21:38:06.689457 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.689101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs\") pod \"19c2f56f-41b1-4275-a865-9b77c129435f\" (UID: \"19c2f56f-41b1-4275-a865-9b77c129435f\") "
Sep 09 21:38:06.691136 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.691107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19c2f56f-41b1-4275-a865-9b77c129435f" (UID: "19c2f56f-41b1-4275-a865-9b77c129435f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:38:06.691206 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.691147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx" (OuterVolumeSpecName: "kube-api-access-wfnnx") pod "19c2f56f-41b1-4275-a865-9b77c129435f" (UID: "19c2f56f-41b1-4275-a865-9b77c129435f"). InnerVolumeSpecName "kube-api-access-wfnnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:38:06.790242 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.790200    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c2f56f-41b1-4275-a865-9b77c129435f-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:38:06.790242 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:06.790239    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnnx\" (UniqueName: \"kubernetes.io/projected/19c2f56f-41b1-4275-a865-9b77c129435f-kube-api-access-wfnnx\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:38:07.561810 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.561768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9" event={"ID":"19c2f56f-41b1-4275-a865-9b77c129435f","Type":"ContainerDied","Data":"60bde64eafb8cafd98ebaf603c990b0e1612a4fd9bad1b6c740e607a32e848f6"}
Sep 09 21:38:07.561986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.561830    2563 scope.go:117] "RemoveContainer" containerID="ececb127c6f8c46fc10e66ec8a8761e608cf92197a290c94bbd8fef73742696a"
Sep 09 21:38:07.561986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.561785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"
Sep 09 21:38:07.588066 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.588040    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:38:07.590805 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.590783    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche92lm9"]
Sep 09 21:38:07.718912 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:07.718878    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" path="/var/lib/kubelet/pods/19c2f56f-41b1-4275-a865-9b77c129435f/volumes"
Sep 09 21:38:27.588529 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588480    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main"
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main"
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588921    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main"
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main"
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.588992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="485a22cb-58be-4151-b30b-0db7b843fca1" containerName="main"
Sep 09 21:38:27.589003 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.589001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c2f56f-41b1-4275-a865-9b77c129435f" containerName="main"
Sep 09 21:38:27.591891 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.591867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.596623 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.596598    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 21:38:27.596921 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.596900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 09 21:38:27.597000 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.596936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 21:38:27.597000 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.596940    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-f2ctg\""
Sep 09 21:38:27.597208 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.597187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-spp4s\""
Sep 09 21:38:27.608259 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.608236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:38:27.768112 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.768080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkzl\" (UniqueName: \"kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.768296 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.768184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.868985 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.868951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkzl\" (UniqueName: \"kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.869169 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.869012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.871549 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.871520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.882878 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.882852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkzl\" (UniqueName: \"kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:27.901798 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:27.901777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:28.180835 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:28.180762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:38:28.184468 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:38:28.184442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed73c81_5180_4a7d_9080_8f926b118d39.slice/crio-b3d8fc1fc7f84ac29faf944f437c7d384f99c1edecd53b9b87a7d3f76e363c9b WatchSource:0}: Error finding container b3d8fc1fc7f84ac29faf944f437c7d384f99c1edecd53b9b87a7d3f76e363c9b: Status 404 returned error can't find the container with id b3d8fc1fc7f84ac29faf944f437c7d384f99c1edecd53b9b87a7d3f76e363c9b
Sep 09 21:38:28.656784 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:28.656752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" event={"ID":"0ed73c81-5180-4a7d-9080-8f926b118d39","Type":"ContainerStarted","Data":"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"}
Sep 09 21:38:28.657168 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:28.656801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" event={"ID":"0ed73c81-5180-4a7d-9080-8f926b118d39","Type":"ContainerStarted","Data":"b3d8fc1fc7f84ac29faf944f437c7d384f99c1edecd53b9b87a7d3f76e363c9b"}
Sep 09 21:38:28.657168 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:28.656883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:38:28.686889 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:28.686840    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podStartSLOduration=1.686819872 podStartE2EDuration="1.686819872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:28.685606845 +0000 UTC m=+2725.437151755" watchObservedRunningTime="2026-09-09 21:38:28.686819872 +0000 UTC m=+2725.438364781"
Sep 09 21:38:59.663463 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:38:59.663430    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:39:07.792626 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.792593    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:39:07.797416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.797391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.800376 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.800354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 09 21:39:07.831499 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.831466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:39:07.892717 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.892870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.892870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.892870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.892870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.892870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.892833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jkj\" (UniqueName: \"kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.993968 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.993942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.993975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jkj\" (UniqueName: \"kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994469 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994566 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.994566 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.994452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.996276 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.996252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:07.996664 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:07.996642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:08.014073 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.014050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jkj\" (UniqueName: \"kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:08.106953 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.106879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:08.132819 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.132790    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:39:08.136650 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.136630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.143372 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.143347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-k9d27\""
Sep 09 21:39:08.152507 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.152467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:39:08.196986 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.196942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zpv\" (UniqueName: \"kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.197114 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.197046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.298102 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.298064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zpv\" (UniqueName: \"kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.298258 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.298115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.300527 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.300507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.311708 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.311689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zpv\" (UniqueName: \"kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.401124 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.401099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:39:08.402405 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:39:08.402384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b2f5f_3a87_4aa5_8db8_2c0ce1539b7a.slice/crio-c22e88d1ea33711f4998a26f141e7c79fe135758800e366b81b422f915f4bc7d WatchSource:0}: Error finding container c22e88d1ea33711f4998a26f141e7c79fe135758800e366b81b422f915f4bc7d: Status 404 returned error can't find the container with id c22e88d1ea33711f4998a26f141e7c79fe135758800e366b81b422f915f4bc7d
Sep 09 21:39:08.404567 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.404547    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:39:08.448126 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.448096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:08.728222 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.728129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:39:08.731638 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:39:08.731611    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d67b9_0587_4c4b_abba_324344af6982.slice/crio-6819d67af48c53fc744f60614a8c7dc3c350d1c2cd37726e419d1c19f3902040 WatchSource:0}: Error finding container 6819d67af48c53fc744f60614a8c7dc3c350d1c2cd37726e419d1c19f3902040: Status 404 returned error can't find the container with id 6819d67af48c53fc744f60614a8c7dc3c350d1c2cd37726e419d1c19f3902040
Sep 09 21:39:08.833759 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.833729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" event={"ID":"bb8d67b9-0587-4c4b-abba-324344af6982","Type":"ContainerStarted","Data":"6819d67af48c53fc744f60614a8c7dc3c350d1c2cd37726e419d1c19f3902040"}
Sep 09 21:39:08.835031 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:08.834992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" event={"ID":"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a","Type":"ContainerStarted","Data":"c22e88d1ea33711f4998a26f141e7c79fe135758800e366b81b422f915f4bc7d"}
Sep 09 21:39:09.840785 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:09.840756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" event={"ID":"bb8d67b9-0587-4c4b-abba-324344af6982","Type":"ContainerStarted","Data":"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"}
Sep 09 21:39:09.841077 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:09.840852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:39:09.862722 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:09.862676    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podStartSLOduration=1.862661422 podStartE2EDuration="1.862661422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:09.861940345 +0000 UTC m=+2766.613485255" watchObservedRunningTime="2026-09-09 21:39:09.862661422 +0000 UTC m=+2766.614206330"
Sep 09 21:39:10.846902 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:10.846849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" event={"ID":"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a","Type":"ContainerStarted","Data":"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"}
Sep 09 21:39:11.108050 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:11.107966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:18.107553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:18.107514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:18.119646 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:18.119619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:18.146470 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:18.146419    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" podStartSLOduration=9.736835753 podStartE2EDuration="11.146406128s" totalImagesPullingTime="1.409570375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:10.880384919 +0000 UTC m=+2767.631929828" watchObservedRunningTime="2026-09-09 21:39:18.146406128 +0000 UTC m=+2774.897951037"
Sep 09 21:39:18.893746 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:18.893718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:39:40.848604 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:39:40.848571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:40:16.050420 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.050390    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:40:16.059531 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:40:16.059480    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:40:16.059531 ip-10-0-138-22 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:40:16.059531 ip-10-0-138-22 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 09 21:40:16.059531 ip-10-0-138-22 kubenswrapper[2563]:  > containerID="472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd" cmd=["/bin/sleep","15"]
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.059543    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" message="" err=<
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:  >
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:40:16.059555    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerName="main" containerID="cri-o://472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"
Sep 09 21:40:16.059768 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.059588    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerName="main" containerID="cri-o://472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd" gracePeriod=60
Sep 09 21:40:16.072224 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.072197    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:40:16.412452 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.412427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:40:16.576920 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.576886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.576941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.576967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.576992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2jkj\" (UniqueName: \"kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm\") pod \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\" (UID: \"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a\") "
Sep 09 21:40:16.577358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:16.577358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:16.577358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577283    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-model-cache\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.577358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577301    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tmp-dir\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.577358 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.577332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home" (OuterVolumeSpecName: "home") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:16.579346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.579287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm" (OuterVolumeSpecName: "dshm") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:40:16.579346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.579293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj" (OuterVolumeSpecName: "kube-api-access-n2jkj") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "kube-api-access-n2jkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:40:16.579346 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.579327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" (UID: "d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:40:16.678625 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.678599    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2jkj\" (UniqueName: \"kubernetes.io/projected/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-kube-api-access-n2jkj\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.678625 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.678624    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-dshm\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.678802 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.678635    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:16.678802 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:16.678648    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a-home\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:40:17.158862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.153998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" event={"ID":"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a","Type":"ContainerDied","Data":"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"}
Sep 09 21:40:17.158862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.154144    2563 scope.go:117] "RemoveContainer" containerID="472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"
Sep 09 21:40:17.158862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.154337    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"
Sep 09 21:40:17.158862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.154372    2563 generic.go:356] "Generic (PLEG): container finished" podID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerID="472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd" exitCode=0
Sep 09 21:40:17.158862 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.154572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls" event={"ID":"d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a","Type":"ContainerDied","Data":"c22e88d1ea33711f4998a26f141e7c79fe135758800e366b81b422f915f4bc7d"}
Sep 09 21:40:17.168741 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.168721    2563 scope.go:117] "RemoveContainer" containerID="472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"
Sep 09 21:40:17.168989 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:40:17.168968    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd\": container with ID starting with 472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd not found: ID does not exist" containerID="472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"
Sep 09 21:40:17.169029 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.168997    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd"} err="failed to get container status \"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd\": rpc error: code = NotFound desc = could not find container \"472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd\": container with ID starting with 472c9633b1be0aff7b1be03023fa0aa6e79833b5f77876ec06b8b3be579dbefd not found: ID does not exist"
Sep 09 21:40:17.184081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.184055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:40:17.191087 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.191061    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sbvls"]
Sep 09 21:40:17.716095 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:17.716052    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" path="/var/lib/kubelet/pods/d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a/volumes"
Sep 09 21:40:20.849230 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:20.849188    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:40:30.849078 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:30.849040    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:40:31.091395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:31.091354    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" containerID="cri-o://12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b" gracePeriod=45
Sep 09 21:40:40.848575 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:40.848534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:40:40.849013 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:40.848663    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:40:50.848880 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:40:50.848834    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:00.848328 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:00.848237    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:01.337882 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.337857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:41:01.347112 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.347087    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb8d67b9-0587-4c4b-abba-324344af6982" containerID="12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b" exitCode=0
Sep 09 21:41:01.347239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.347136    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"
Sep 09 21:41:01.347239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.347168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" event={"ID":"bb8d67b9-0587-4c4b-abba-324344af6982","Type":"ContainerDied","Data":"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"}
Sep 09 21:41:01.347239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.347218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz" event={"ID":"bb8d67b9-0587-4c4b-abba-324344af6982","Type":"ContainerDied","Data":"6819d67af48c53fc744f60614a8c7dc3c350d1c2cd37726e419d1c19f3902040"}
Sep 09 21:41:01.347363 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.347242    2563 scope.go:117] "RemoveContainer" containerID="12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"
Sep 09 21:41:01.354894 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.354869    2563 scope.go:117] "RemoveContainer" containerID="12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"
Sep 09 21:41:01.355183 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:41:01.355140    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b\": container with ID starting with 12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b not found: ID does not exist" containerID="12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"
Sep 09 21:41:01.355183 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.355169    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b"} err="failed to get container status \"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b\": rpc error: code = NotFound desc = could not find container \"12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b\": container with ID starting with 12fcf11c584b1aacbbe964bce04f4b848c3dfe8e91fac588ff01b87d82c7aa2b not found: ID does not exist"
Sep 09 21:41:01.440175 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.440146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs\") pod \"bb8d67b9-0587-4c4b-abba-324344af6982\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") "
Sep 09 21:41:01.440332 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.440190    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8zpv\" (UniqueName: \"kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv\") pod \"bb8d67b9-0587-4c4b-abba-324344af6982\" (UID: \"bb8d67b9-0587-4c4b-abba-324344af6982\") "
Sep 09 21:41:01.442179 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.442141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb8d67b9-0587-4c4b-abba-324344af6982" (UID: "bb8d67b9-0587-4c4b-abba-324344af6982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:01.442300 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.442190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv" (OuterVolumeSpecName: "kube-api-access-s8zpv") pod "bb8d67b9-0587-4c4b-abba-324344af6982" (UID: "bb8d67b9-0587-4c4b-abba-324344af6982"). InnerVolumeSpecName "kube-api-access-s8zpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:01.541612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.541579    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8d67b9-0587-4c4b-abba-324344af6982-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:41:01.541612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.541607    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zpv\" (UniqueName: \"kubernetes.io/projected/bb8d67b9-0587-4c4b-abba-324344af6982-kube-api-access-s8zpv\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:41:01.671890 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.671859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:41:01.676140 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.676114    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schewnkxz"]
Sep 09 21:41:01.715103 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:01.715070    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" path="/var/lib/kubelet/pods/bb8d67b9-0587-4c4b-abba-324344af6982/volumes"
Sep 09 21:41:03.861102 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:03.861068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:41:09.663780 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:09.663743    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:18.885305 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:18.885242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main" containerID="cri-o://25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314" gracePeriod=45
Sep 09 21:41:19.663585 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:19.663548    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:29.663906 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:29.663860    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:29.664399 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:29.663967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:41:39.663760 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:39.663721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:41:49.124420 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.124395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:41:49.203796 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.203730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs\") pod \"0ed73c81-5180-4a7d-9080-8f926b118d39\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") "
Sep 09 21:41:49.203958 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.203825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwkzl\" (UniqueName: \"kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl\") pod \"0ed73c81-5180-4a7d-9080-8f926b118d39\" (UID: \"0ed73c81-5180-4a7d-9080-8f926b118d39\") "
Sep 09 21:41:49.205800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.205766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl" (OuterVolumeSpecName: "kube-api-access-cwkzl") pod "0ed73c81-5180-4a7d-9080-8f926b118d39" (UID: "0ed73c81-5180-4a7d-9080-8f926b118d39"). InnerVolumeSpecName "kube-api-access-cwkzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:49.205910 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.205799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ed73c81-5180-4a7d-9080-8f926b118d39" (UID: "0ed73c81-5180-4a7d-9080-8f926b118d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:49.304710 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.304682    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkzl\" (UniqueName: \"kubernetes.io/projected/0ed73c81-5180-4a7d-9080-8f926b118d39-kube-api-access-cwkzl\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:41:49.304710 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.304707    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed73c81-5180-4a7d-9080-8f926b118d39-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:41:49.560881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.560789    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerID="25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314" exitCode=0
Sep 09 21:41:49.560881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.560834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"
Sep 09 21:41:49.560881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.560861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" event={"ID":"0ed73c81-5180-4a7d-9080-8f926b118d39","Type":"ContainerDied","Data":"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"}
Sep 09 21:41:49.561127 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.560908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk" event={"ID":"0ed73c81-5180-4a7d-9080-8f926b118d39","Type":"ContainerDied","Data":"b3d8fc1fc7f84ac29faf944f437c7d384f99c1edecd53b9b87a7d3f76e363c9b"}
Sep 09 21:41:49.561127 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.560933    2563 scope.go:117] "RemoveContainer" containerID="25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"
Sep 09 21:41:49.569535 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.569518    2563 scope.go:117] "RemoveContainer" containerID="25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"
Sep 09 21:41:49.569778 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:41:49.569759    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314\": container with ID starting with 25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314 not found: ID does not exist" containerID="25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"
Sep 09 21:41:49.569844 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.569784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314"} err="failed to get container status \"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314\": rpc error: code = NotFound desc = could not find container \"25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314\": container with ID starting with 25a8a2e464c2e53b367259bb79fc1329db2bc140c4f0a1500b161280280b8314 not found: ID does not exist"
Sep 09 21:41:49.581918 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.581896    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:41:49.587737 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.587716    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schenvpzk"]
Sep 09 21:41:49.715257 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:49.715222    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" path="/var/lib/kubelet/pods/0ed73c81-5180-4a7d-9080-8f926b118d39/volumes"
Sep 09 21:41:59.593230 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593195    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:41:59.593722 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main"
Sep 09 21:41:59.593794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main"
Sep 09 21:41:59.593794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593742    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerName="main"
Sep 09 21:41:59.593794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593752    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerName="main"
Sep 09 21:41:59.593794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main"
Sep 09 21:41:59.593794 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593786    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main"
Sep 09 21:41:59.594049 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593860    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8d67b9-0587-4c4b-abba-324344af6982" containerName="main"
Sep 09 21:41:59.594049 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593874    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed73c81-5180-4a7d-9080-8f926b118d39" containerName="main"
Sep 09 21:41:59.594049 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.593886    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56b2f5f-3a87-4aa5-8db8-2c0ce1539b7a" containerName="main"
Sep 09 21:41:59.598298 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.598274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.602345 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.602320    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 09 21:41:59.602478 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.602441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-mr546\""
Sep 09 21:41:59.602576 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.602527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-kggcb\""
Sep 09 21:41:59.602643 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.602578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 09 21:41:59.602697 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.602643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 09 21:41:59.607704 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.607683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:41:59.688467 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.688434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.688654 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.688481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhzj\" (UniqueName: \"kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.789117 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.789087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.789276 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.789138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhzj\" (UniqueName: \"kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.791600 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.791576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.799021 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.798991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhzj\" (UniqueName: \"kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:41:59.909577 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:41:59.909547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:42:00.181132 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:00.181107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:42:00.183690 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:42:00.183662    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f8521b_b455_46f6_8e95_f2d3d218f7a3.slice/crio-b4b4d25474822703cac5523f5e1a6b66e1af8e4e6e5babf11ffec4013c85679b WatchSource:0}: Error finding container b4b4d25474822703cac5523f5e1a6b66e1af8e4e6e5babf11ffec4013c85679b: Status 404 returned error can't find the container with id b4b4d25474822703cac5523f5e1a6b66e1af8e4e6e5babf11ffec4013c85679b
Sep 09 21:42:00.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:00.613262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" event={"ID":"d1f8521b-b455-46f6-8e95-f2d3d218f7a3","Type":"ContainerStarted","Data":"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"}
Sep 09 21:42:00.613350 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:00.613322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" event={"ID":"d1f8521b-b455-46f6-8e95-f2d3d218f7a3","Type":"ContainerStarted","Data":"b4b4d25474822703cac5523f5e1a6b66e1af8e4e6e5babf11ffec4013c85679b"}
Sep 09 21:42:00.614165 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:00.613472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:42:00.632856 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:00.632808    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podStartSLOduration=1.6327938309999999 podStartE2EDuration="1.632793831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:00.631097462 +0000 UTC m=+2937.382642372" watchObservedRunningTime="2026-09-09 21:42:00.632793831 +0000 UTC m=+2937.384338740"
Sep 09 21:42:31.621770 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:42:31.621695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:43:03.864801 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:03.864771    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:43:03.873688 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:03.873660    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:43:21.121081 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.121046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:43:21.124693 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.124673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.128859 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.128829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 09 21:43:21.128859 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.128839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-86tn8\""
Sep 09 21:43:21.129060 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.128839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 09 21:43:21.129060 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.128888    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-bfz4n\""
Sep 09 21:43:21.129060 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.128864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 09 21:43:21.132564 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.132543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:43:21.236239 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.236200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.236425 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.236248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6j9\" (UniqueName: \"kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.336997 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.336962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.337146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.337009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6j9\" (UniqueName: \"kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.339446 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.339424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.349549 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.349516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6j9\" (UniqueName: \"kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.436156 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.436087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.703466 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.703441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:43:21.705834 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:43:21.705804    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8203bf_8851_4030_90b6_10ba7c5a1979.slice/crio-72f062ed839653465908d0ac6910ee81c98e49b1bacdac76162aa11e54b761de WatchSource:0}: Error finding container 72f062ed839653465908d0ac6910ee81c98e49b1bacdac76162aa11e54b761de: Status 404 returned error can't find the container with id 72f062ed839653465908d0ac6910ee81c98e49b1bacdac76162aa11e54b761de
Sep 09 21:43:21.977269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.977166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" event={"ID":"8f8203bf-8851-4030-90b6-10ba7c5a1979","Type":"ContainerStarted","Data":"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"}
Sep 09 21:43:21.977269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.977232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" event={"ID":"8f8203bf-8851-4030-90b6-10ba7c5a1979","Type":"ContainerStarted","Data":"72f062ed839653465908d0ac6910ee81c98e49b1bacdac76162aa11e54b761de"}
Sep 09 21:43:21.977269 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.977259    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:43:21.995843 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:21.995794    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podStartSLOduration=0.995778573 podStartE2EDuration="995.778573ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:43:21.99364945 +0000 UTC m=+3018.745194360" watchObservedRunningTime="2026-09-09 21:43:21.995778573 +0000 UTC m=+3018.747323482"
Sep 09 21:43:48.245552 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:48.245510    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:43:51.621825 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:51.621787    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:43:52.983985 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:43:52.983945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:44:01.621873 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:01.621786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:44:03.265235 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:03.265199    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" containerID="cri-o://2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea" gracePeriod=45
Sep 09 21:44:11.621791 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:11.621749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:44:11.622356 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:11.621883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:44:21.621500 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:21.621445    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:44:31.621236 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:31.621198    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:44:33.520222 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.520196    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:44:33.642411 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.642383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs\") pod \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") "
Sep 09 21:44:33.642579 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.642438    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xhzj\" (UniqueName: \"kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj\") pod \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\" (UID: \"d1f8521b-b455-46f6-8e95-f2d3d218f7a3\") "
Sep 09 21:44:33.644707 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.644678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj" (OuterVolumeSpecName: "kube-api-access-9xhzj") pod "d1f8521b-b455-46f6-8e95-f2d3d218f7a3" (UID: "d1f8521b-b455-46f6-8e95-f2d3d218f7a3"). InnerVolumeSpecName "kube-api-access-9xhzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:33.644823 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.644678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1f8521b-b455-46f6-8e95-f2d3d218f7a3" (UID: "d1f8521b-b455-46f6-8e95-f2d3d218f7a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:33.743444 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.743415    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:44:33.743444 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:33.743442    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhzj\" (UniqueName: \"kubernetes.io/projected/d1f8521b-b455-46f6-8e95-f2d3d218f7a3-kube-api-access-9xhzj\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:44:34.305359 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.305324    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerID="2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea" exitCode=0
Sep 09 21:44:34.305543 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.305368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"
Sep 09 21:44:34.305543 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.305377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" event={"ID":"d1f8521b-b455-46f6-8e95-f2d3d218f7a3","Type":"ContainerDied","Data":"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"}
Sep 09 21:44:34.305543 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.305416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9" event={"ID":"d1f8521b-b455-46f6-8e95-f2d3d218f7a3","Type":"ContainerDied","Data":"b4b4d25474822703cac5523f5e1a6b66e1af8e4e6e5babf11ffec4013c85679b"}
Sep 09 21:44:34.305543 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.305436    2563 scope.go:117] "RemoveContainer" containerID="2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"
Sep 09 21:44:34.313582 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.313558    2563 scope.go:117] "RemoveContainer" containerID="2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"
Sep 09 21:44:34.313841 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:44:34.313822    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea\": container with ID starting with 2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea not found: ID does not exist" containerID="2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"
Sep 09 21:44:34.313898 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.313849    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea"} err="failed to get container status \"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea\": rpc error: code = NotFound desc = could not find container \"2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea\": container with ID starting with 2429f3453f1d2f6097ec52f377046a3df36eef27236bf2ac7e070b4b7e437cea not found: ID does not exist"
Sep 09 21:44:34.322638 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.322618    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:44:34.326005 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:34.325985    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche257r9"]
Sep 09 21:44:35.715685 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:35.715653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" path="/var/lib/kubelet/pods/d1f8521b-b455-46f6-8e95-f2d3d218f7a3/volumes"
Sep 09 21:44:58.989734 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.989698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:44:58.990251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.990034    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main"
Sep 09 21:44:58.990251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.990046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main"
Sep 09 21:44:58.990251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.990110    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f8521b-b455-46f6-8e95-f2d3d218f7a3" containerName="main"
Sep 09 21:44:58.993113 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.993083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:58.997036 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.997011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-c4qw2\""
Sep 09 21:44:58.997150 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.997058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 09 21:44:58.997150 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.997103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9fj8t\""
Sep 09 21:44:58.997150 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.997122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 09 21:44:58.997322 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:58.997058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 09 21:44:59.003747 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.003728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:44:59.044047 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.044015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.044166 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.044053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8ds\" (UniqueName: \"kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.144590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.144559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.144741 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.144601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8ds\" (UniqueName: \"kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.147148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.147119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.153237 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.153207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8ds\" (UniqueName: \"kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.304797 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.304702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:44:59.574925 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:44:59.574842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:44:59.577027 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:44:59.576989    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9265bc71_6411_4c26_8c89_822a8916d0c1.slice/crio-e0cac38d0da8320df301ac8d25d8e310543b578b8ae8387994840d44db9168f8 WatchSource:0}: Error finding container e0cac38d0da8320df301ac8d25d8e310543b578b8ae8387994840d44db9168f8: Status 404 returned error can't find the container with id e0cac38d0da8320df301ac8d25d8e310543b578b8ae8387994840d44db9168f8
Sep 09 21:45:00.421322 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:00.421278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" event={"ID":"9265bc71-6411-4c26-8c89-822a8916d0c1","Type":"ContainerStarted","Data":"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"}
Sep 09 21:45:00.421748 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:00.421330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" event={"ID":"9265bc71-6411-4c26-8c89-822a8916d0c1","Type":"ContainerStarted","Data":"e0cac38d0da8320df301ac8d25d8e310543b578b8ae8387994840d44db9168f8"}
Sep 09 21:45:00.421748 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:00.421422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:45:00.439084 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:00.439018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podStartSLOduration=2.439001153 podStartE2EDuration="2.439001153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:00.438293409 +0000 UTC m=+3117.189838343" watchObservedRunningTime="2026-09-09 21:45:00.439001153 +0000 UTC m=+3117.190546062"
Sep 09 21:45:31.427617 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:31.427528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:45:35.720345 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:35.720308    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:45:42.984338 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:42.984296    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:45:50.740641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:50.740570    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main" containerID="cri-o://f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b" gracePeriod=45
Sep 09 21:45:52.983846 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:45:52.983807    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:02.984460 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:02.984418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:02.984964 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:02.984546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:46:12.983914 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:12.983870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:20.619844 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:20.619805    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:46:21.000099 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.000075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:46:21.105873 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.105833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs\") pod \"8f8203bf-8851-4030-90b6-10ba7c5a1979\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") "
Sep 09 21:46:21.106083 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.105929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb6j9\" (UniqueName: \"kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9\") pod \"8f8203bf-8851-4030-90b6-10ba7c5a1979\" (UID: \"8f8203bf-8851-4030-90b6-10ba7c5a1979\") "
Sep 09 21:46:21.108171 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.108139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9" (OuterVolumeSpecName: "kube-api-access-nb6j9") pod "8f8203bf-8851-4030-90b6-10ba7c5a1979" (UID: "8f8203bf-8851-4030-90b6-10ba7c5a1979"). InnerVolumeSpecName "kube-api-access-nb6j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:21.108276 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.108211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f8203bf-8851-4030-90b6-10ba7c5a1979" (UID: "8f8203bf-8851-4030-90b6-10ba7c5a1979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:21.206556 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.206403    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6j9\" (UniqueName: \"kubernetes.io/projected/8f8203bf-8851-4030-90b6-10ba7c5a1979-kube-api-access-nb6j9\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:46:21.206556 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.206437    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8203bf-8851-4030-90b6-10ba7c5a1979-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:46:21.427339 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.427298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:21.774745 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.774706    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerID="f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b" exitCode=0
Sep 09 21:46:21.775237 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.774760    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"
Sep 09 21:46:21.775237 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.774785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" event={"ID":"8f8203bf-8851-4030-90b6-10ba7c5a1979","Type":"ContainerDied","Data":"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"}
Sep 09 21:46:21.775237 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.774839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc" event={"ID":"8f8203bf-8851-4030-90b6-10ba7c5a1979","Type":"ContainerDied","Data":"72f062ed839653465908d0ac6910ee81c98e49b1bacdac76162aa11e54b761de"}
Sep 09 21:46:21.775237 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.774868    2563 scope.go:117] "RemoveContainer" containerID="f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"
Sep 09 21:46:21.783212 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.783193    2563 scope.go:117] "RemoveContainer" containerID="f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"
Sep 09 21:46:21.783514 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:46:21.783481    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b\": container with ID starting with f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b not found: ID does not exist" containerID="f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"
Sep 09 21:46:21.783582 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.783527    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b"} err="failed to get container status \"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b\": rpc error: code = NotFound desc = could not find container \"f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b\": container with ID starting with f4fea417b627a4ac24cc337323ef47f764a723fed97d8c7573227fe9c5443a8b not found: ID does not exist"
Sep 09 21:46:21.791539 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.791513    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:46:21.794699 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:21.794677    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegjlrc"]
Sep 09 21:46:23.715225 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:23.715191    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" path="/var/lib/kubelet/pods/8f8203bf-8851-4030-90b6-10ba7c5a1979/volumes"
Sep 09 21:46:31.427884 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:31.427846    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:32.184384 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.184340    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:46:32.184814 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.184795    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main"
Sep 09 21:46:32.184928 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.184816    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main"
Sep 09 21:46:32.184928 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.184904    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8203bf-8851-4030-90b6-10ba7c5a1979" containerName="main"
Sep 09 21:46:32.189443 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.189421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.193148 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.193123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 09 21:46:32.193367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.193177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-n2vvp\""
Sep 09 21:46:32.193367 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.193177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-nfzcp\""
Sep 09 21:46:32.193562 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.193215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 09 21:46:32.193562 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.193255    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 09 21:46:32.198055 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.198000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:46:32.306322 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.306282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr42t\" (UniqueName: \"kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.306513 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.306393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.407695 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.407653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr42t\" (UniqueName: \"kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.407866 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.407737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.410266 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.410247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.416410 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.416373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr42t\" (UniqueName: \"kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.502298 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.502207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:32.775356 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.775280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:46:32.779146 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:46:32.779117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69abd17d_a736_4b4b_98ca_4562f10f7a73.slice/crio-262aadb8eeb8a7ca749454c86e52287fd47c8f9edc00ed5c518af6de2783f4ba WatchSource:0}: Error finding container 262aadb8eeb8a7ca749454c86e52287fd47c8f9edc00ed5c518af6de2783f4ba: Status 404 returned error can't find the container with id 262aadb8eeb8a7ca749454c86e52287fd47c8f9edc00ed5c518af6de2783f4ba
Sep 09 21:46:32.824251 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:32.824218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" event={"ID":"69abd17d-a736-4b4b-98ca-4562f10f7a73","Type":"ContainerStarted","Data":"262aadb8eeb8a7ca749454c86e52287fd47c8f9edc00ed5c518af6de2783f4ba"}
Sep 09 21:46:33.829870 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:33.829832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" event={"ID":"69abd17d-a736-4b4b-98ca-4562f10f7a73","Type":"ContainerStarted","Data":"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"}
Sep 09 21:46:33.830324 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:33.829900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:46:33.847131 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:33.847082    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podStartSLOduration=1.847065873 podStartE2EDuration="1.847065873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:33.846031278 +0000 UTC m=+3210.597576187" watchObservedRunningTime="2026-09-09 21:46:33.847065873 +0000 UTC m=+3210.598610782"
Sep 09 21:46:35.641114 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:35.641073    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" containerID="cri-o://989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea" gracePeriod=45
Sep 09 21:46:41.427887 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:41.427847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:46:41.428322 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:41.427967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:46:51.427714 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:46:51.427670    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:47:01.428064 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:01.427978    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:47:04.837802 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:04.837765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:47:05.892097 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.892066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:47:05.977181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.977092    2563 generic.go:356] "Generic (PLEG): container finished" podID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerID="989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea" exitCode=0
Sep 09 21:47:05.977181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.977142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"
Sep 09 21:47:05.977380 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.977175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" event={"ID":"9265bc71-6411-4c26-8c89-822a8916d0c1","Type":"ContainerDied","Data":"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"}
Sep 09 21:47:05.977380 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.977233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd" event={"ID":"9265bc71-6411-4c26-8c89-822a8916d0c1","Type":"ContainerDied","Data":"e0cac38d0da8320df301ac8d25d8e310543b578b8ae8387994840d44db9168f8"}
Sep 09 21:47:05.977380 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.977260    2563 scope.go:117] "RemoveContainer" containerID="989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"
Sep 09 21:47:05.985205 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.985185    2563 scope.go:117] "RemoveContainer" containerID="989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"
Sep 09 21:47:05.985460 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:47:05.985439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea\": container with ID starting with 989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea not found: ID does not exist" containerID="989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"
Sep 09 21:47:05.985570 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.985470    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea"} err="failed to get container status \"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea\": rpc error: code = NotFound desc = could not find container \"989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea\": container with ID starting with 989850f996c43169c1e76da091c8bf2ada08a8106c71c20247bb5acabfe4b1ea not found: ID does not exist"
Sep 09 21:47:05.993220 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.993203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs\") pod \"9265bc71-6411-4c26-8c89-822a8916d0c1\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") "
Sep 09 21:47:05.993310 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.993249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k8ds\" (UniqueName: \"kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds\") pod \"9265bc71-6411-4c26-8c89-822a8916d0c1\" (UID: \"9265bc71-6411-4c26-8c89-822a8916d0c1\") "
Sep 09 21:47:05.995361 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.995332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9265bc71-6411-4c26-8c89-822a8916d0c1" (UID: "9265bc71-6411-4c26-8c89-822a8916d0c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:05.995448 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:05.995380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds" (OuterVolumeSpecName: "kube-api-access-9k8ds") pod "9265bc71-6411-4c26-8c89-822a8916d0c1" (UID: "9265bc71-6411-4c26-8c89-822a8916d0c1"). InnerVolumeSpecName "kube-api-access-9k8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:06.094051 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:06.094011    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8ds\" (UniqueName: \"kubernetes.io/projected/9265bc71-6411-4c26-8c89-822a8916d0c1-kube-api-access-9k8ds\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:47:06.094051 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:06.094046    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9265bc71-6411-4c26-8c89-822a8916d0c1-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:47:06.300266 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:06.300234    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:47:06.303981 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:06.303955    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schex9dvd"]
Sep 09 21:47:07.715799 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:07.715751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" path="/var/lib/kubelet/pods/9265bc71-6411-4c26-8c89-822a8916d0c1/volumes"
Sep 09 21:47:26.580074 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.580037    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:47:26.580669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.580513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main"
Sep 09 21:47:26.580669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.580534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main"
Sep 09 21:47:26.580669 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.580639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9265bc71-6411-4c26-8c89-822a8916d0c1" containerName="main"
Sep 09 21:47:26.583749 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.583727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.588438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.588210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 09 21:47:26.588438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.588269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 09 21:47:26.588438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.588337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-qqw2c\""
Sep 09 21:47:26.588438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.588365    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 09 21:47:26.588808 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.588785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-tmm9m\""
Sep 09 21:47:26.593934 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.593911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:47:26.662360 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.662326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzsdl\" (UniqueName: \"kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.662528 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.662368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.763602 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.763568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzsdl\" (UniqueName: \"kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.763766 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.763622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.766152 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.766128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.771379 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.771348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzsdl\" (UniqueName: \"kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:26.895467 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:26.895437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:27.165982 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:27.165948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:47:27.168387 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:47:27.168357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf9f543_eae8_46a7_946a_24a10571f83c.slice/crio-7ae525ba7169b35da176c246048e4fa322fbb6b1d0e05247ec6e115b7ad00364 WatchSource:0}: Error finding container 7ae525ba7169b35da176c246048e4fa322fbb6b1d0e05247ec6e115b7ad00364: Status 404 returned error can't find the container with id 7ae525ba7169b35da176c246048e4fa322fbb6b1d0e05247ec6e115b7ad00364
Sep 09 21:47:28.080589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:28.080552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" event={"ID":"6cf9f543-eae8-46a7-946a-24a10571f83c","Type":"ContainerStarted","Data":"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"}
Sep 09 21:47:28.081007 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:28.080608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" event={"ID":"6cf9f543-eae8-46a7-946a-24a10571f83c","Type":"ContainerStarted","Data":"7ae525ba7169b35da176c246048e4fa322fbb6b1d0e05247ec6e115b7ad00364"}
Sep 09 21:47:28.081007 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:28.080630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:47:28.098884 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:28.098840    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" podStartSLOduration=2.0988259830000002 podStartE2EDuration="2.098825983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:28.09697199 +0000 UTC m=+3264.848516899" watchObservedRunningTime="2026-09-09 21:47:28.098825983 +0000 UTC m=+3264.850370936"
Sep 09 21:47:30.399223 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:30.399186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:47:45.421771 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:45.421731    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main" containerID="cri-o://1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11" gracePeriod=45
Sep 09 21:47:46.694830 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.694798    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:47:46.698225 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.698201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.701424 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.701402    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"e2e-del-unblock-svc-epp-sa-dockercfg-s4dsk\""
Sep 09 21:47:46.701562 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.701450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"kube-root-ca.crt\""
Sep 09 21:47:46.703019 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.702992    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"openshift-service-ca.crt\""
Sep 09 21:47:46.703149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.702992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"default-dockercfg-k4pxq\""
Sep 09 21:47:46.703149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.703053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"e2e-del-unblock-svc-kserve-self-signed-certs\""
Sep 09 21:47:46.710762 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.710743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:47:46.831191 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.831145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txnq7\" (UniqueName: \"kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.831395 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.831220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.932542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.932479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txnq7\" (UniqueName: \"kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.932542 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.932546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.935067 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.935040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:46.942920 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:46.942894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txnq7\" (UniqueName: \"kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7\") pod \"e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:47.010703 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:47.010629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:47.290528 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:47.290505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:47:47.292931 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:47:47.292898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873d9702_cac3_4cd7_b65a_0797a236c360.slice/crio-24b140c00d003e79e0318b62a61d299892395b0524b4cf37cf90f89930fba007 WatchSource:0}: Error finding container 24b140c00d003e79e0318b62a61d299892395b0524b4cf37cf90f89930fba007: Status 404 returned error can't find the container with id 24b140c00d003e79e0318b62a61d299892395b0524b4cf37cf90f89930fba007
Sep 09 21:47:48.166694 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:48.166656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" event={"ID":"873d9702-cac3-4cd7-b65a-0797a236c360","Type":"ContainerStarted","Data":"cd42732279b4b31262bc31fe9b6c46d6fd19691f90c24dd3f813764f9554d888"}
Sep 09 21:47:48.167135 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:48.166714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" event={"ID":"873d9702-cac3-4cd7-b65a-0797a236c360","Type":"ContainerStarted","Data":"24b140c00d003e79e0318b62a61d299892395b0524b4cf37cf90f89930fba007"}
Sep 09 21:47:48.167135 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:48.166736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:47:48.187990 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:48.187945    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" podStartSLOduration=2.187929693 podStartE2EDuration="2.187929693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:48.186110479 +0000 UTC m=+3284.937655387" watchObservedRunningTime="2026-09-09 21:47:48.187929693 +0000 UTC m=+3284.939474613"
Sep 09 21:47:49.944876 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:47:49.944841    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:48:00.403082 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:00.403041    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:03.898604 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:03.898577    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:48:03.904911 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:03.904881    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:48:05.193505 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:05.193445    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main" containerID="cri-o://cd42732279b4b31262bc31fe9b6c46d6fd19691f90c24dd3f813764f9554d888" gracePeriod=45
Sep 09 21:48:10.401769 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:10.401728    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:15.658676 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.658647    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:48:15.774957 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.774889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs\") pod \"6cf9f543-eae8-46a7-946a-24a10571f83c\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") "
Sep 09 21:48:15.775100 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.774978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzsdl\" (UniqueName: \"kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl\") pod \"6cf9f543-eae8-46a7-946a-24a10571f83c\" (UID: \"6cf9f543-eae8-46a7-946a-24a10571f83c\") "
Sep 09 21:48:15.777039 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.777003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cf9f543-eae8-46a7-946a-24a10571f83c" (UID: "6cf9f543-eae8-46a7-946a-24a10571f83c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:15.777154 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.777052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl" (OuterVolumeSpecName: "kube-api-access-bzsdl") pod "6cf9f543-eae8-46a7-946a-24a10571f83c" (UID: "6cf9f543-eae8-46a7-946a-24a10571f83c"). InnerVolumeSpecName "kube-api-access-bzsdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:15.876316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.876286    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzsdl\" (UniqueName: \"kubernetes.io/projected/6cf9f543-eae8-46a7-946a-24a10571f83c-kube-api-access-bzsdl\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:48:15.876316 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:15.876311    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cf9f543-eae8-46a7-946a-24a10571f83c-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:48:16.293006 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.292968    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerID="1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11" exitCode=0
Sep 09 21:48:16.293178 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.293013    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"
Sep 09 21:48:16.293178 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.293046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" event={"ID":"6cf9f543-eae8-46a7-946a-24a10571f83c","Type":"ContainerDied","Data":"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"}
Sep 09 21:48:16.293178 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.293096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh" event={"ID":"6cf9f543-eae8-46a7-946a-24a10571f83c","Type":"ContainerDied","Data":"7ae525ba7169b35da176c246048e4fa322fbb6b1d0e05247ec6e115b7ad00364"}
Sep 09 21:48:16.293178 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.293119    2563 scope.go:117] "RemoveContainer" containerID="1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"
Sep 09 21:48:16.302349 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.302332    2563 scope.go:117] "RemoveContainer" containerID="1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"
Sep 09 21:48:16.302623 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:48:16.302602    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11\": container with ID starting with 1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11 not found: ID does not exist" containerID="1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"
Sep 09 21:48:16.302687 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.302639    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11"} err="failed to get container status \"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11\": rpc error: code = NotFound desc = could not find container \"1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11\": container with ID starting with 1e281b80d72473d80591733d262ec8a1de138cdee6b55ddd9cdb391b37bcdb11 not found: ID does not exist"
Sep 09 21:48:16.315280 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.315252    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:48:16.317511 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:16.317472    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-8vtdh"]
Sep 09 21:48:17.716647 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:17.716612    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" path="/var/lib/kubelet/pods/6cf9f543-eae8-46a7-946a-24a10571f83c/volumes"
Sep 09 21:48:18.040374 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.040301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:48:18.040829 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.040812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main"
Sep 09 21:48:18.040878 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.040833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main"
Sep 09 21:48:18.040929 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.040919    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf9f543-eae8-46a7-946a-24a10571f83c" containerName="main"
Sep 09 21:48:18.045828 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.045643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.050646 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.050606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 21:48:18.051151 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.051130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-lnjzv\""
Sep 09 21:48:18.051826 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.051792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 21:48:18.052008 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.051954    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 21:48:18.052143 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.051883    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-hv7pw\""
Sep 09 21:48:18.053893 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.053873    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:48:18.090923 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.090886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrss\" (UniqueName: \"kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.091080 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.090937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.191884 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.191845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrss\" (UniqueName: \"kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.192020 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.191887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.194418 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.194387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.201295 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.201273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrss\" (UniqueName: \"kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.358881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.358795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:18.630025 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:18.629990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:48:18.632655 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:48:18.632627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983ef2cd_22ef_42f9_86a3_2349e74a3db3.slice/crio-fa172081e0674339637a4c1f5b064f815d1b290be18b65c2a11cf0bb36cdb1de WatchSource:0}: Error finding container fa172081e0674339637a4c1f5b064f815d1b290be18b65c2a11cf0bb36cdb1de: Status 404 returned error can't find the container with id fa172081e0674339637a4c1f5b064f815d1b290be18b65c2a11cf0bb36cdb1de
Sep 09 21:48:19.311044 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:19.311012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" event={"ID":"983ef2cd-22ef-42f9-86a3-2349e74a3db3","Type":"ContainerStarted","Data":"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"}
Sep 09 21:48:19.311450 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:19.311061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" event={"ID":"983ef2cd-22ef-42f9-86a3-2349e74a3db3","Type":"ContainerStarted","Data":"fa172081e0674339637a4c1f5b064f815d1b290be18b65c2a11cf0bb36cdb1de"}
Sep 09 21:48:19.311450 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:19.311148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:19.331434 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:19.331376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podStartSLOduration=1.331355979 podStartE2EDuration="1.331355979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:19.330596107 +0000 UTC m=+3316.082141018" watchObservedRunningTime="2026-09-09 21:48:19.331355979 +0000 UTC m=+3316.082900888"
Sep 09 21:48:20.177590 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:20.177546    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:30.175800 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:30.175717    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:48:35.388142 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.388104    2563 generic.go:356] "Generic (PLEG): container finished" podID="873d9702-cac3-4cd7-b65a-0797a236c360" containerID="cd42732279b4b31262bc31fe9b6c46d6fd19691f90c24dd3f813764f9554d888" exitCode=0
Sep 09 21:48:35.388475 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.388168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" event={"ID":"873d9702-cac3-4cd7-b65a-0797a236c360","Type":"ContainerDied","Data":"cd42732279b4b31262bc31fe9b6c46d6fd19691f90c24dd3f813764f9554d888"}
Sep 09 21:48:35.441078 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.441048    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:48:35.535392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.535313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txnq7\" (UniqueName: \"kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7\") pod \"873d9702-cac3-4cd7-b65a-0797a236c360\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") "
Sep 09 21:48:35.535392 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.535386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs\") pod \"873d9702-cac3-4cd7-b65a-0797a236c360\" (UID: \"873d9702-cac3-4cd7-b65a-0797a236c360\") "
Sep 09 21:48:35.537547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.537515    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "873d9702-cac3-4cd7-b65a-0797a236c360" (UID: "873d9702-cac3-4cd7-b65a-0797a236c360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:48:35.537547 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.537524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7" (OuterVolumeSpecName: "kube-api-access-txnq7") pod "873d9702-cac3-4cd7-b65a-0797a236c360" (UID: "873d9702-cac3-4cd7-b65a-0797a236c360"). InnerVolumeSpecName "kube-api-access-txnq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:35.636134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.636096    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d9702-cac3-4cd7-b65a-0797a236c360-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:48:35.636134 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:35.636130    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txnq7\" (UniqueName: \"kubernetes.io/projected/873d9702-cac3-4cd7-b65a-0797a236c360-kube-api-access-txnq7\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:48:36.394162 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:36.394131    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"
Sep 09 21:48:36.394642 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:36.394172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc" event={"ID":"873d9702-cac3-4cd7-b65a-0797a236c360","Type":"ContainerDied","Data":"24b140c00d003e79e0318b62a61d299892395b0524b4cf37cf90f89930fba007"}
Sep 09 21:48:36.394642 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:36.394235    2563 scope.go:117] "RemoveContainer" containerID="cd42732279b4b31262bc31fe9b6c46d6fd19691f90c24dd3f813764f9554d888"
Sep 09 21:48:36.410676 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:36.410651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:48:36.413728 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:36.413708    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-router-scheduler-7cbc5f97bc-qwddc"]
Sep 09 21:48:37.722248 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:37.722210    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" path="/var/lib/kubelet/pods/873d9702-cac3-4cd7-b65a-0797a236c360/volumes"
Sep 09 21:48:47.142962 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:47.142927    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:48:50.320837 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:50.320805    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:48:54.837349 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:48:54.837309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:49:02.162994 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:02.162928    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main" containerID="cri-o://c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72" gracePeriod=45
Sep 09 21:49:04.837476 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:04.837436    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:49:14.837438 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:14.837397    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:49:14.837841 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:14.837530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:49:24.837697 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:24.837661    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:49:32.405051 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.405024    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:49:32.499155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.499074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs\") pod \"69abd17d-a736-4b4b-98ca-4562f10f7a73\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") "
Sep 09 21:49:32.499155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.499132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr42t\" (UniqueName: \"kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t\") pod \"69abd17d-a736-4b4b-98ca-4562f10f7a73\" (UID: \"69abd17d-a736-4b4b-98ca-4562f10f7a73\") "
Sep 09 21:49:32.501291 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.501262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69abd17d-a736-4b4b-98ca-4562f10f7a73" (UID: "69abd17d-a736-4b4b-98ca-4562f10f7a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:49:32.501397 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.501291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t" (OuterVolumeSpecName: "kube-api-access-xr42t") pod "69abd17d-a736-4b4b-98ca-4562f10f7a73" (UID: "69abd17d-a736-4b4b-98ca-4562f10f7a73"). InnerVolumeSpecName "kube-api-access-xr42t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:32.600287 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.600257    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69abd17d-a736-4b4b-98ca-4562f10f7a73-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:49:32.600287 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.600284    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr42t\" (UniqueName: \"kubernetes.io/projected/69abd17d-a736-4b4b-98ca-4562f10f7a73-kube-api-access-xr42t\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:49:32.650930 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.650895    2563 generic.go:356] "Generic (PLEG): container finished" podID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerID="c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72" exitCode=0
Sep 09 21:49:32.651061 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.650942    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"
Sep 09 21:49:32.651061 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.650971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" event={"ID":"69abd17d-a736-4b4b-98ca-4562f10f7a73","Type":"ContainerDied","Data":"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"}
Sep 09 21:49:32.651061 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.651013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb" event={"ID":"69abd17d-a736-4b4b-98ca-4562f10f7a73","Type":"ContainerDied","Data":"262aadb8eeb8a7ca749454c86e52287fd47c8f9edc00ed5c518af6de2783f4ba"}
Sep 09 21:49:32.651061 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.651034    2563 scope.go:117] "RemoveContainer" containerID="c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"
Sep 09 21:49:32.659384 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.659361    2563 scope.go:117] "RemoveContainer" containerID="c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"
Sep 09 21:49:32.659667 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:49:32.659647    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72\": container with ID starting with c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72 not found: ID does not exist" containerID="c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"
Sep 09 21:49:32.659741 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.659676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72"} err="failed to get container status \"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72\": rpc error: code = NotFound desc = could not find container \"c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72\": container with ID starting with c9bc81984aee0170f4e674cf425c56efaa0d5b27aca70bf8d022b9d4062b6f72 not found: ID does not exist"
Sep 09 21:49:32.671383 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.671344    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:49:32.673536 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:32.673515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589cntb"]
Sep 09 21:49:33.714657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:33.714624    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" path="/var/lib/kubelet/pods/69abd17d-a736-4b4b-98ca-4562f10f7a73/volumes"
Sep 09 21:49:54.110608 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.110575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:49:54.111042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.110960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main"
Sep 09 21:49:54.111042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.110974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main"
Sep 09 21:49:54.111042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.110988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main"
Sep 09 21:49:54.111042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.110994    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main"
Sep 09 21:49:54.111184 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.111050    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69abd17d-a736-4b4b-98ca-4562f10f7a73" containerName="main"
Sep 09 21:49:54.111184 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.111060    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="873d9702-cac3-4cd7-b65a-0797a236c360" containerName="main"
Sep 09 21:49:54.114032 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.114003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.116641 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.116615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 21:49:54.117782 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.117759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 09 21:49:54.117958 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.117930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-nzfp4\""
Sep 09 21:49:54.118042 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.117995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 21:49:54.118125 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.118073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-9xlzm\""
Sep 09 21:49:54.122992 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.122963    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:49:54.173326 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.173299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.173432 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.173331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h794g\" (UniqueName: \"kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.273943 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.273911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.274071 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.273952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h794g\" (UniqueName: \"kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.276345 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.276321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.281729 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.281703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h794g\" (UniqueName: \"kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.424808 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.424779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:54.697759 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.697733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:49:54.700180 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:49:54.700148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652ee6ab_0377_4b55_8518_6350413cf4fd.slice/crio-5405463b9e5ea380a33e3e45420a9f8736a1ecff61f477b2651289dad2cd6281 WatchSource:0}: Error finding container 5405463b9e5ea380a33e3e45420a9f8736a1ecff61f477b2651289dad2cd6281: Status 404 returned error can't find the container with id 5405463b9e5ea380a33e3e45420a9f8736a1ecff61f477b2651289dad2cd6281
Sep 09 21:49:54.753131 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:54.753098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" event={"ID":"652ee6ab-0377-4b55-8518-6350413cf4fd","Type":"ContainerStarted","Data":"5405463b9e5ea380a33e3e45420a9f8736a1ecff61f477b2651289dad2cd6281"}
Sep 09 21:49:55.758472 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:55.758433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" event={"ID":"652ee6ab-0377-4b55-8518-6350413cf4fd","Type":"ContainerStarted","Data":"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"}
Sep 09 21:49:55.758895 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:55.758516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:49:55.777386 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:49:55.777338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podStartSLOduration=1.777323948 podStartE2EDuration="1.777323948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:55.775164007 +0000 UTC m=+3412.526708917" watchObservedRunningTime="2026-09-09 21:49:55.777323948 +0000 UTC m=+3412.528868860"
Sep 09 21:50:20.001130 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:20.001092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:50:20.320788 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:20.320707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:50:26.765555 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:26.765523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:50:35.024424 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:35.024383    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main" containerID="cri-o://8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184" gracePeriod=45
Sep 09 21:50:40.321219 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:40.321178    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:50:50.321105 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:50:50.321066    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:51:00.321399 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:00.321358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:51:00.321904 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:00.321512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:51:05.269256 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.269231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:51:05.359847 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.359780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs\") pod \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") "
Sep 09 21:51:05.359961 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.359877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtrss\" (UniqueName: \"kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss\") pod \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\" (UID: \"983ef2cd-22ef-42f9-86a3-2349e74a3db3\") "
Sep 09 21:51:05.361931 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.361905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "983ef2cd-22ef-42f9-86a3-2349e74a3db3" (UID: "983ef2cd-22ef-42f9-86a3-2349e74a3db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:51:05.362050 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.361940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss" (OuterVolumeSpecName: "kube-api-access-vtrss") pod "983ef2cd-22ef-42f9-86a3-2349e74a3db3" (UID: "983ef2cd-22ef-42f9-86a3-2349e74a3db3"). InnerVolumeSpecName "kube-api-access-vtrss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:51:05.460456 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.460417    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983ef2cd-22ef-42f9-86a3-2349e74a3db3-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:51:05.460456 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:05.460453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrss\" (UniqueName: \"kubernetes.io/projected/983ef2cd-22ef-42f9-86a3-2349e74a3db3-kube-api-access-vtrss\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:51:06.070181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.070093    2563 generic.go:356] "Generic (PLEG): container finished" podID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerID="8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184" exitCode=0
Sep 09 21:51:06.070181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.070137    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"
Sep 09 21:51:06.070181 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.070169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" event={"ID":"983ef2cd-22ef-42f9-86a3-2349e74a3db3","Type":"ContainerDied","Data":"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"}
Sep 09 21:51:06.070436 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.070209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc" event={"ID":"983ef2cd-22ef-42f9-86a3-2349e74a3db3","Type":"ContainerDied","Data":"fa172081e0674339637a4c1f5b064f815d1b290be18b65c2a11cf0bb36cdb1de"}
Sep 09 21:51:06.070436 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.070230    2563 scope.go:117] "RemoveContainer" containerID="8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"
Sep 09 21:51:06.078535 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.078517    2563 scope.go:117] "RemoveContainer" containerID="8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"
Sep 09 21:51:06.078803 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:51:06.078783    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184\": container with ID starting with 8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184 not found: ID does not exist" containerID="8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"
Sep 09 21:51:06.078877 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.078810    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184"} err="failed to get container status \"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184\": rpc error: code = NotFound desc = could not find container \"8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184\": container with ID starting with 8594b8000c21aeefd6ca35d23e5c4dec88b15431ca2dec290513cce676af3184 not found: ID does not exist"
Sep 09 21:51:06.087704 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.087680    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:51:06.089105 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:06.089086    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nxdlc"]
Sep 09 21:51:07.716421 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:07.716372    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" path="/var/lib/kubelet/pods/983ef2cd-22ef-42f9-86a3-2349e74a3db3/volumes"
Sep 09 21:51:58.553155 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:51:58.553119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:52:06.765719 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:06.765674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:13.572831 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:13.572771    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main" containerID="cri-o://ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313" gracePeriod=45
Sep 09 21:52:16.765279 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:16.765241    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:26.765528 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:26.765465    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:26.765987 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:26.765587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:52:36.765573 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:36.765528    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:52:43.823196 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.823169    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:52:43.854856 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.854787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs\") pod \"652ee6ab-0377-4b55-8518-6350413cf4fd\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") "
Sep 09 21:52:43.854856 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.854830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h794g\" (UniqueName: \"kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g\") pod \"652ee6ab-0377-4b55-8518-6350413cf4fd\" (UID: \"652ee6ab-0377-4b55-8518-6350413cf4fd\") "
Sep 09 21:52:43.856977 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.856948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "652ee6ab-0377-4b55-8518-6350413cf4fd" (UID: "652ee6ab-0377-4b55-8518-6350413cf4fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:52:43.857104 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.857027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g" (OuterVolumeSpecName: "kube-api-access-h794g") pod "652ee6ab-0377-4b55-8518-6350413cf4fd" (UID: "652ee6ab-0377-4b55-8518-6350413cf4fd"). InnerVolumeSpecName "kube-api-access-h794g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:52:43.955721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.955686    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652ee6ab-0377-4b55-8518-6350413cf4fd-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:52:43.955721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:43.955717    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h794g\" (UniqueName: \"kubernetes.io/projected/652ee6ab-0377-4b55-8518-6350413cf4fd-kube-api-access-h794g\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:52:44.501424 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.501386    2563 generic.go:356] "Generic (PLEG): container finished" podID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerID="ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313" exitCode=0
Sep 09 21:52:44.501612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.501434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"
Sep 09 21:52:44.501612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.501454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" event={"ID":"652ee6ab-0377-4b55-8518-6350413cf4fd","Type":"ContainerDied","Data":"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"}
Sep 09 21:52:44.501612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.501533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj" event={"ID":"652ee6ab-0377-4b55-8518-6350413cf4fd","Type":"ContainerDied","Data":"5405463b9e5ea380a33e3e45420a9f8736a1ecff61f477b2651289dad2cd6281"}
Sep 09 21:52:44.501612 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.501564    2563 scope.go:117] "RemoveContainer" containerID="ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"
Sep 09 21:52:44.509883 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.509860    2563 scope.go:117] "RemoveContainer" containerID="ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"
Sep 09 21:52:44.510210 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:52:44.510189    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313\": container with ID starting with ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313 not found: ID does not exist" containerID="ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"
Sep 09 21:52:44.510278 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.510219    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313"} err="failed to get container status \"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313\": rpc error: code = NotFound desc = could not find container \"ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313\": container with ID starting with ed028719cd173b52f82b356c88537c7d6cf99c606774734d327fe281c7587313 not found: ID does not exist"
Sep 09 21:52:44.523143 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.523120    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:52:44.526823 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:44.526803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-66vzj"]
Sep 09 21:52:45.715597 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:45.715562    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" path="/var/lib/kubelet/pods/652ee6ab-0377-4b55-8518-6350413cf4fd/volumes"
Sep 09 21:52:54.314084 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main"
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main"
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314444    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main"
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314450    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main"
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314522    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="652ee6ab-0377-4b55-8518-6350413cf4fd" containerName="main"
Sep 09 21:52:54.314541 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.314532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="983ef2cd-22ef-42f9-86a3-2349e74a3db3" containerName="main"
Sep 09 21:52:54.317435 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.317413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.321146 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.321120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-94gvt\""
Sep 09 21:52:54.321266 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.321132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 21:52:54.321266 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.321191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-kbmx8\""
Sep 09 21:52:54.321266 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.321204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 21:52:54.321461 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.321340    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 21:52:54.328353 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.328329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:52:54.328631 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.328611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpvp\" (UniqueName: \"kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.328739 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.328719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.430131 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.430090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.430303 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.430159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpvp\" (UniqueName: \"kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.432665 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.432642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.437987 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.437962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpvp\" (UniqueName: \"kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.628752 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.628726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:54.899645 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:54.899619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:52:54.901941 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:52:54.901911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dd9b9e_3191_4c30_84d0_ac1d784c550f.slice/crio-e92631a6388198264ae2a0aa460d6a7012ca34d3ec5cf30c9ca0cccb237a8bb6 WatchSource:0}: Error finding container e92631a6388198264ae2a0aa460d6a7012ca34d3ec5cf30c9ca0cccb237a8bb6: Status 404 returned error can't find the container with id e92631a6388198264ae2a0aa460d6a7012ca34d3ec5cf30c9ca0cccb237a8bb6
Sep 09 21:52:55.554093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:55.554044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" event={"ID":"e0dd9b9e-3191-4c30-84d0-ac1d784c550f","Type":"ContainerStarted","Data":"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"}
Sep 09 21:52:55.554589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:55.554106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" event={"ID":"e0dd9b9e-3191-4c30-84d0-ac1d784c550f","Type":"ContainerStarted","Data":"e92631a6388198264ae2a0aa460d6a7012ca34d3ec5cf30c9ca0cccb237a8bb6"}
Sep 09 21:52:55.554589 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:55.554132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:52:55.571098 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:52:55.571054    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podStartSLOduration=1.57103908 podStartE2EDuration="1.57103908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:55.56996578 +0000 UTC m=+3592.321510702" watchObservedRunningTime="2026-09-09 21:52:55.57103908 +0000 UTC m=+3592.322583989"
Sep 09 21:53:03.925085 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:53:03.925047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:53:03.932473 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:53:03.932448    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:53:26.560735 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:53:26.560696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:55:56.217462 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:55:56.217371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:55:56.561766 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:55:56.561648    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:56:06.560938 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:06.560894    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:56:11.239191 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:11.239134    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" containerID="cri-o://6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586" gracePeriod=45
Sep 09 21:56:16.561463 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:16.561420    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:56:16.561972 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:16.561559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:56:26.560711 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:26.560674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:56:36.561326 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:36.561287    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:56:41.477416 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.477392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:56:41.537553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.537455    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerID="6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586" exitCode=0
Sep 09 21:56:41.537553 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.537512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"
Sep 09 21:56:41.537721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.537540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" event={"ID":"e0dd9b9e-3191-4c30-84d0-ac1d784c550f","Type":"ContainerDied","Data":"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"}
Sep 09 21:56:41.537721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.537606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2" event={"ID":"e0dd9b9e-3191-4c30-84d0-ac1d784c550f","Type":"ContainerDied","Data":"e92631a6388198264ae2a0aa460d6a7012ca34d3ec5cf30c9ca0cccb237a8bb6"}
Sep 09 21:56:41.537721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.537639    2563 scope.go:117] "RemoveContainer" containerID="6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"
Sep 09 21:56:41.545734 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.545718    2563 scope.go:117] "RemoveContainer" containerID="6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"
Sep 09 21:56:41.545995 ip-10-0-138-22 kubenswrapper[2563]: E0909 21:56:41.545976    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586\": container with ID starting with 6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586 not found: ID does not exist" containerID="6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"
Sep 09 21:56:41.546076 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.546003    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586"} err="failed to get container status \"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586\": rpc error: code = NotFound desc = could not find container \"6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586\": container with ID starting with 6a3522783cdb3e7f4fb0fdb84134a27b483dcd15859afcb7311c75bf4613a586 not found: ID does not exist"
Sep 09 21:56:41.557881 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.557858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgpvp\" (UniqueName: \"kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp\") pod \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") "
Sep 09 21:56:41.557980 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.557890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs\") pod \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\" (UID: \"e0dd9b9e-3191-4c30-84d0-ac1d784c550f\") "
Sep 09 21:56:41.560086 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.560061    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp" (OuterVolumeSpecName: "kube-api-access-xgpvp") pod "e0dd9b9e-3191-4c30-84d0-ac1d784c550f" (UID: "e0dd9b9e-3191-4c30-84d0-ac1d784c550f"). InnerVolumeSpecName "kube-api-access-xgpvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:41.560173 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.560081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0dd9b9e-3191-4c30-84d0-ac1d784c550f" (UID: "e0dd9b9e-3191-4c30-84d0-ac1d784c550f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:56:41.658375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.658355    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpvp\" (UniqueName: \"kubernetes.io/projected/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-kube-api-access-xgpvp\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:56:41.658375 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.658375    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0dd9b9e-3191-4c30-84d0-ac1d784c550f-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 21:56:41.856706 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.856625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:56:41.861940 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:41.861907    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche2bvv2"]
Sep 09 21:56:43.715149 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:43.715116    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" path="/var/lib/kubelet/pods/e0dd9b9e-3191-4c30-84d0-ac1d784c550f/volumes"
Sep 09 21:56:52.841390 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.841354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 21:56:52.841779 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.841703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main"
Sep 09 21:56:52.841779 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.841715    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main"
Sep 09 21:56:52.841857 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.841786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dd9b9e-3191-4c30-84d0-ac1d784c550f" containerName="main"
Sep 09 21:56:52.844546 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.844529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:52.849473 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.849449    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 21:56:52.849635 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.849469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-c4t85\""
Sep 09 21:56:52.849635 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.849463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 21:56:52.849756 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.849538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-mv4mr\""
Sep 09 21:56:52.849756 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.849541    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 21:56:52.880806 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.880784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 21:56:52.950568 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.950477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:52.950700 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:52.950585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvtx\" (UniqueName: \"kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.051551 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.051512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.051734 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.051558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvtx\" (UniqueName: \"kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.054218 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.054192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.061514 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.061472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvtx\" (UniqueName: \"kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.154766 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.154735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.431851 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.431824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 21:56:53.433831 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:56:53.433803    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7609b39_81cf_4798_9922_5aa1e1a79503.slice/crio-a1d180fa6c7036b563e72f3a039782d7ac72ec5fe24355f370f59b6fec2ec3f9 WatchSource:0}: Error finding container a1d180fa6c7036b563e72f3a039782d7ac72ec5fe24355f370f59b6fec2ec3f9: Status 404 returned error can't find the container with id a1d180fa6c7036b563e72f3a039782d7ac72ec5fe24355f370f59b6fec2ec3f9
Sep 09 21:56:53.602070 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.602032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" event={"ID":"e7609b39-81cf-4798-9922-5aa1e1a79503","Type":"ContainerStarted","Data":"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"}
Sep 09 21:56:53.602279 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.602083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" event={"ID":"e7609b39-81cf-4798-9922-5aa1e1a79503","Type":"ContainerStarted","Data":"a1d180fa6c7036b563e72f3a039782d7ac72ec5fe24355f370f59b6fec2ec3f9"}
Sep 09 21:56:53.602279 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.602151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:56:53.626175 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:56:53.626124    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podStartSLOduration=1.626108188 podStartE2EDuration="1.626108188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:53.624463949 +0000 UTC m=+3830.376008859" watchObservedRunningTime="2026-09-09 21:56:53.626108188 +0000 UTC m=+3830.377653097"
Sep 09 21:57:24.608721 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:57:24.608690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 21:58:03.950730 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:03.950695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:58:03.958058 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:03.958022    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 21:58:23.219094 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.219060    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 21:58:23.222377 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.222350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.226032 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.225999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 09 21:58:23.227332 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.227302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 09 21:58:23.227473 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.227453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 09 21:58:23.227587 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.227560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-8fczr\""
Sep 09 21:58:23.227722 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.227706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-8wjff\""
Sep 09 21:58:23.235965 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.235945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 21:58:23.264080 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.264058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.264190 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.264113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qs6z\" (UniqueName: \"kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.365009 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.364975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.365158 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.365081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qs6z\" (UniqueName: \"kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.367651 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.367620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.374093 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.374070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qs6z\" (UniqueName: \"kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.534224 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.534141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:23.805657 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.805624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 21:58:23.808208 ip-10-0-138-22 kubenswrapper[2563]: W0909 21:58:23.808177    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f389005_1c3a_4261_9752_e6ec730f23d8.slice/crio-1f79de012d2114ccb09314a46ce1e9a44a3d8c283234f165fa213e9931f1c37b WatchSource:0}: Error finding container 1f79de012d2114ccb09314a46ce1e9a44a3d8c283234f165fa213e9931f1c37b: Status 404 returned error can't find the container with id 1f79de012d2114ccb09314a46ce1e9a44a3d8c283234f165fa213e9931f1c37b
Sep 09 21:58:23.988812 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.988780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" event={"ID":"5f389005-1c3a-4261-9752-e6ec730f23d8","Type":"ContainerStarted","Data":"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"}
Sep 09 21:58:23.989010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.988828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" event={"ID":"5f389005-1c3a-4261-9752-e6ec730f23d8","Type":"ContainerStarted","Data":"1f79de012d2114ccb09314a46ce1e9a44a3d8c283234f165fa213e9931f1c37b"}
Sep 09 21:58:23.989010 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:23.988889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:58:24.012389 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:24.012338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podStartSLOduration=1.012323219 podStartE2EDuration="1.012323219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:58:24.009675933 +0000 UTC m=+3920.761220841" watchObservedRunningTime="2026-09-09 21:58:24.012323219 +0000 UTC m=+3920.763868152"
Sep 09 21:58:54.994724 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:58:54.994686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 21:59:41.924777 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:59:41.924742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 21:59:44.608564 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:59:44.608526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:59:54.608741 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:59:54.608692    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:59:56.944583 ip-10-0-138-22 kubenswrapper[2563]: I0909 21:59:56.944541    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" containerID="cri-o://ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8" gracePeriod=45
Sep 09 22:00:04.609015 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:04.608973    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:04.609405 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:04.609084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 22:00:14.608906 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:14.608865    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:24.608458 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:24.608416    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:27.186995 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.186971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 22:00:27.269837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.269768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mvtx\" (UniqueName: \"kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx\") pod \"e7609b39-81cf-4798-9922-5aa1e1a79503\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") "
Sep 09 22:00:27.269837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.269815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs\") pod \"e7609b39-81cf-4798-9922-5aa1e1a79503\" (UID: \"e7609b39-81cf-4798-9922-5aa1e1a79503\") "
Sep 09 22:00:27.271919 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.271895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx" (OuterVolumeSpecName: "kube-api-access-2mvtx") pod "e7609b39-81cf-4798-9922-5aa1e1a79503" (UID: "e7609b39-81cf-4798-9922-5aa1e1a79503"). InnerVolumeSpecName "kube-api-access-2mvtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:00:27.272030 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.271916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7609b39-81cf-4798-9922-5aa1e1a79503" (UID: "e7609b39-81cf-4798-9922-5aa1e1a79503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:00:27.370329 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.370284    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvtx\" (UniqueName: \"kubernetes.io/projected/e7609b39-81cf-4798-9922-5aa1e1a79503-kube-api-access-2mvtx\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.370329 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.370320    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7609b39-81cf-4798-9922-5aa1e1a79503-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:00:27.542076 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.541984    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerID="ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8" exitCode=0
Sep 09 22:00:27.542076 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.542035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"
Sep 09 22:00:27.542076 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.542065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" event={"ID":"e7609b39-81cf-4798-9922-5aa1e1a79503","Type":"ContainerDied","Data":"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"}
Sep 09 22:00:27.542297 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.542114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j" event={"ID":"e7609b39-81cf-4798-9922-5aa1e1a79503","Type":"ContainerDied","Data":"a1d180fa6c7036b563e72f3a039782d7ac72ec5fe24355f370f59b6fec2ec3f9"}
Sep 09 22:00:27.542297 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.542136    2563 scope.go:117] "RemoveContainer" containerID="ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"
Sep 09 22:00:27.550657 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.550642    2563 scope.go:117] "RemoveContainer" containerID="ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"
Sep 09 22:00:27.550874 ip-10-0-138-22 kubenswrapper[2563]: E0909 22:00:27.550855    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8\": container with ID starting with ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8 not found: ID does not exist" containerID="ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"
Sep 09 22:00:27.550921 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.550880    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8"} err="failed to get container status \"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8\": rpc error: code = NotFound desc = could not find container \"ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8\": container with ID starting with ab0dce60a180812e7e4188dc1eee600b4a147d2fe0b5d63fdf238951d67378c8 not found: ID does not exist"
Sep 09 22:00:27.564417 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.564398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 22:00:27.568510 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.568476    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48s7j8j"]
Sep 09 22:00:27.715542 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:27.715509    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" path="/var/lib/kubelet/pods/e7609b39-81cf-4798-9922-5aa1e1a79503/volumes"
Sep 09 22:00:28.141841 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:28.141809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 22:00:34.995528 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:34.995467    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:43.167031 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:43.166985    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main" containerID="cri-o://d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb" gracePeriod=45
Sep 09 22:00:44.994852 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:44.994804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:45.555462 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.555428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:00:45.555779 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.555766    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main"
Sep 09 22:00:45.555827 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.555781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main"
Sep 09 22:00:45.555867 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.555833    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7609b39-81cf-4798-9922-5aa1e1a79503" containerName="main"
Sep 09 22:00:45.560140 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.560118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.564018 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.563991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 22:00:45.564144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.564020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 22:00:45.564144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.564029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6xgvz\""
Sep 09 22:00:45.564144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.564005    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 22:00:45.564144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.564019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-bzgk6\""
Sep 09 22:00:45.570370 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.570348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:00:45.608693 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.608661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.608819 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.608714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwqg\" (UniqueName: \"kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.709860 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.709826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.710063 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.709881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwqg\" (UniqueName: \"kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.712674 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.712639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.723954 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.723920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwqg\" (UniqueName: \"kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:45.870858 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:45.870824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:46.148234 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:46.148128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:00:46.150763 ip-10-0-138-22 kubenswrapper[2563]: W0909 22:00:46.150736    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae2cc9a_b652_46d0_93ea_7a48bbc19438.slice/crio-05cc366fdc4eccfb3a5f37c22f1c2947c5803f2b5c952c592dbc9afddf1adf69 WatchSource:0}: Error finding container 05cc366fdc4eccfb3a5f37c22f1c2947c5803f2b5c952c592dbc9afddf1adf69: Status 404 returned error can't find the container with id 05cc366fdc4eccfb3a5f37c22f1c2947c5803f2b5c952c592dbc9afddf1adf69
Sep 09 22:00:46.629980 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:46.629944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" event={"ID":"eae2cc9a-b652-46d0-93ea-7a48bbc19438","Type":"ContainerStarted","Data":"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"}
Sep 09 22:00:46.630155 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:46.629999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" event={"ID":"eae2cc9a-b652-46d0-93ea-7a48bbc19438","Type":"ContainerStarted","Data":"05cc366fdc4eccfb3a5f37c22f1c2947c5803f2b5c952c592dbc9afddf1adf69"}
Sep 09 22:00:46.630155 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:46.630031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:00:46.648901 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:46.648855    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podStartSLOduration=1.648839953 podStartE2EDuration="1.648839953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:46.647576663 +0000 UTC m=+4063.399121575" watchObservedRunningTime="2026-09-09 22:00:46.648839953 +0000 UTC m=+4063.400384862"
Sep 09 22:00:54.995670 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:54.995628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:00:54.996054 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:00:54.995742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 22:01:04.995007 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:04.994962    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:01:13.420334 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.420310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 22:01:13.546831 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.546747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qs6z\" (UniqueName: \"kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z\") pod \"5f389005-1c3a-4261-9752-e6ec730f23d8\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") "
Sep 09 22:01:13.546975 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.546833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs\") pod \"5f389005-1c3a-4261-9752-e6ec730f23d8\" (UID: \"5f389005-1c3a-4261-9752-e6ec730f23d8\") "
Sep 09 22:01:13.549067 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.549038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f389005-1c3a-4261-9752-e6ec730f23d8" (UID: "5f389005-1c3a-4261-9752-e6ec730f23d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:01:13.549067 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.549044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z" (OuterVolumeSpecName: "kube-api-access-4qs6z") pod "5f389005-1c3a-4261-9752-e6ec730f23d8" (UID: "5f389005-1c3a-4261-9752-e6ec730f23d8"). InnerVolumeSpecName "kube-api-access-4qs6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:01:13.647600 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.647554    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f389005-1c3a-4261-9752-e6ec730f23d8-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:01:13.647600 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.647592    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qs6z\" (UniqueName: \"kubernetes.io/projected/5f389005-1c3a-4261-9752-e6ec730f23d8-kube-api-access-4qs6z\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:01:13.749694 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.749661    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerID="d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb" exitCode=0
Sep 09 22:01:13.749837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.749701    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"
Sep 09 22:01:13.749837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.749714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" event={"ID":"5f389005-1c3a-4261-9752-e6ec730f23d8","Type":"ContainerDied","Data":"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"}
Sep 09 22:01:13.749837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.749753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx" event={"ID":"5f389005-1c3a-4261-9752-e6ec730f23d8","Type":"ContainerDied","Data":"1f79de012d2114ccb09314a46ce1e9a44a3d8c283234f165fa213e9931f1c37b"}
Sep 09 22:01:13.749837 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.749774    2563 scope.go:117] "RemoveContainer" containerID="d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"
Sep 09 22:01:13.759775 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.759753    2563 scope.go:117] "RemoveContainer" containerID="d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"
Sep 09 22:01:13.760047 ip-10-0-138-22 kubenswrapper[2563]: E0909 22:01:13.760026    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb\": container with ID starting with d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb not found: ID does not exist" containerID="d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"
Sep 09 22:01:13.760092 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.760055    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb"} err="failed to get container status \"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb\": rpc error: code = NotFound desc = could not find container \"d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb\": container with ID starting with d5c96b4befd2c5fdb1a8eddf85e470790e854e69923db85639aa2d54cbad31cb not found: ID does not exist"
Sep 09 22:01:13.767896 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.767864    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 22:01:13.771318 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:13.771299    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-79pqx"]
Sep 09 22:01:15.715397 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:15.715363    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" path="/var/lib/kubelet/pods/5f389005-1c3a-4261-9752-e6ec730f23d8/volumes"
Sep 09 22:01:17.637568 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:17.637538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:01:30.640392 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.640357    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:01:30.640823 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.640716    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main"
Sep 09 22:01:30.640823 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.640728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main"
Sep 09 22:01:30.640823 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.640807    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f389005-1c3a-4261-9752-e6ec730f23d8" containerName="main"
Sep 09 22:01:30.643917 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.643896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.647869 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.647838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 22:01:30.648006 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.647898    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 22:01:30.648006 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.647944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 22:01:30.648006 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.647900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-l7ccz\""
Sep 09 22:01:30.648511 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.648165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nhv94\""
Sep 09 22:01:30.654315 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.654292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:01:30.677359 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.677335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn94q\" (UniqueName: \"kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.677455 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.677371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.778704 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.778668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn94q\" (UniqueName: \"kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.778838 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.778711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.781224 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.781203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.787248 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.787224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn94q\" (UniqueName: \"kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:30.955119 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:30.955046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:31.224751 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:31.224722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:01:31.227922 ip-10-0-138-22 kubenswrapper[2563]: W0909 22:01:31.227894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d135ee_8fdd_44d7_844e_a8a09fe0ea36.slice/crio-f5845387ff1234577f5f792ddc9ef585341ae529d949978a7a895570d4e6878a WatchSource:0}: Error finding container f5845387ff1234577f5f792ddc9ef585341ae529d949978a7a895570d4e6878a: Status 404 returned error can't find the container with id f5845387ff1234577f5f792ddc9ef585341ae529d949978a7a895570d4e6878a
Sep 09 22:01:31.825890 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:31.825856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" event={"ID":"71d135ee-8fdd-44d7-844e-a8a09fe0ea36","Type":"ContainerStarted","Data":"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"}
Sep 09 22:01:31.826371 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:31.825912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" event={"ID":"71d135ee-8fdd-44d7-844e-a8a09fe0ea36","Type":"ContainerStarted","Data":"f5845387ff1234577f5f792ddc9ef585341ae529d949978a7a895570d4e6878a"}
Sep 09 22:01:31.826371 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:31.825940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:01:31.845704 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:01:31.845656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podStartSLOduration=1.84563998 podStartE2EDuration="1.84563998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:31.845407857 +0000 UTC m=+4108.596952772" watchObservedRunningTime="2026-09-09 22:01:31.84563998 +0000 UTC m=+4108.597184890"
Sep 09 22:02:02.832799 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:02:02.832710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:02:57.932861 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:02:57.932826    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:03:03.975734 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:03.975703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 22:03:03.990425 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:03.990398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 22:03:07.637362 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:07.637314    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:03:12.953211 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:12.953151    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main" containerID="cri-o://8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10" gracePeriod=45
Sep 09 22:03:17.637618 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:17.637572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:03:27.637611 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:27.637516    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:03:27.637986 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:27.637636    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:03:37.638111 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:37.638073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:03:43.205920 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.205895    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:03:43.278928 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.278857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs\") pod \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") "
Sep 09 22:03:43.279054 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.278934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxwqg\" (UniqueName: \"kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg\") pod \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\" (UID: \"eae2cc9a-b652-46d0-93ea-7a48bbc19438\") "
Sep 09 22:03:43.281018 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.280989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eae2cc9a-b652-46d0-93ea-7a48bbc19438" (UID: "eae2cc9a-b652-46d0-93ea-7a48bbc19438"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:43.281132 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.281057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg" (OuterVolumeSpecName: "kube-api-access-kxwqg") pod "eae2cc9a-b652-46d0-93ea-7a48bbc19438" (UID: "eae2cc9a-b652-46d0-93ea-7a48bbc19438"). InnerVolumeSpecName "kube-api-access-kxwqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:43.380119 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.380087    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae2cc9a-b652-46d0-93ea-7a48bbc19438-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:03:43.380119 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.380115    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwqg\" (UniqueName: \"kubernetes.io/projected/eae2cc9a-b652-46d0-93ea-7a48bbc19438-kube-api-access-kxwqg\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:03:43.406970 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.406937    2563 generic.go:356] "Generic (PLEG): container finished" podID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerID="8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10" exitCode=0
Sep 09 22:03:43.407085 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.406989    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"
Sep 09 22:03:43.407085 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.407004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" event={"ID":"eae2cc9a-b652-46d0-93ea-7a48bbc19438","Type":"ContainerDied","Data":"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"}
Sep 09 22:03:43.407085 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.407056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k" event={"ID":"eae2cc9a-b652-46d0-93ea-7a48bbc19438","Type":"ContainerDied","Data":"05cc366fdc4eccfb3a5f37c22f1c2947c5803f2b5c952c592dbc9afddf1adf69"}
Sep 09 22:03:43.407193 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.407086    2563 scope.go:117] "RemoveContainer" containerID="8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"
Sep 09 22:03:43.416833 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.416815    2563 scope.go:117] "RemoveContainer" containerID="8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"
Sep 09 22:03:43.417142 ip-10-0-138-22 kubenswrapper[2563]: E0909 22:03:43.417123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10\": container with ID starting with 8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10 not found: ID does not exist" containerID="8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"
Sep 09 22:03:43.417194 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.417151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10"} err="failed to get container status \"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10\": rpc error: code = NotFound desc = could not find container \"8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10\": container with ID starting with 8f05630245243d8b9689a97169a3407600099e89c01df6ae0b9287b46cd56d10 not found: ID does not exist"
Sep 09 22:03:43.436849 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.436824    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:03:43.441238 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.441215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45qxs9k"]
Sep 09 22:03:43.715216 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:03:43.715179    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" path="/var/lib/kubelet/pods/eae2cc9a-b652-46d0-93ea-7a48bbc19438/volumes"
Sep 09 22:04:16.962216 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.962175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:04:16.962679 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.962553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main"
Sep 09 22:04:16.962679 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.962566    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main"
Sep 09 22:04:16.962679 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.962621    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae2cc9a-b652-46d0-93ea-7a48bbc19438" containerName="main"
Sep 09 22:04:16.966760 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.966737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:16.970679 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 22:04:16.970801 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 22:04:16.970872 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-l8bsj\""
Sep 09 22:04:16.970872 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-nh5dm\""
Sep 09 22:04:16.970872 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 22:04:16.970872 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:16.971081 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.970906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2fk\" (UniqueName: \"kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:16.975681 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:16.975650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:04:17.071636 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.071601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2fk\" (UniqueName: \"kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:17.071822 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.071677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:17.074137 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.074116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:17.082198 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.082175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2fk\" (UniqueName: \"kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:17.277721 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.277642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:17.552137 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.551987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:04:17.555548 ip-10-0-138-22 kubenswrapper[2563]: W0909 22:04:17.555519    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902a62b1_705e_4ad1_93af_ecd33ff117c7.slice/crio-92e91b7b7ddf6f2f8ab6de6d9a619eb37eaf4611f3f57b2621cfc0ab8c44887b WatchSource:0}: Error finding container 92e91b7b7ddf6f2f8ab6de6d9a619eb37eaf4611f3f57b2621cfc0ab8c44887b: Status 404 returned error can't find the container with id 92e91b7b7ddf6f2f8ab6de6d9a619eb37eaf4611f3f57b2621cfc0ab8c44887b
Sep 09 22:04:17.565376 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:17.565349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" event={"ID":"902a62b1-705e-4ad1-93af-ecd33ff117c7","Type":"ContainerStarted","Data":"92e91b7b7ddf6f2f8ab6de6d9a619eb37eaf4611f3f57b2621cfc0ab8c44887b"}
Sep 09 22:04:18.571293 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:18.571257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" event={"ID":"902a62b1-705e-4ad1-93af-ecd33ff117c7","Type":"ContainerStarted","Data":"f3bba18e0bea73031d1e0519f4b9fb98a79f1925da267106f29457cb9021cb22"}
Sep 09 22:04:18.571760 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:18.571353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:18.593006 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:18.592936    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podStartSLOduration=2.592920383 podStartE2EDuration="2.592920383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:18.591937389 +0000 UTC m=+4275.343482308" watchObservedRunningTime="2026-09-09 22:04:18.592920383 +0000 UTC m=+4275.344465292"
Sep 09 22:04:21.991015 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:21.990983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:04:22.832331 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:22.832290    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:04:32.832427 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:32.832379    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:04:37.011832 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:37.011763    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" containerID="cri-o://63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d" gracePeriod=45
Sep 09 22:04:42.832264 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:42.832227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:04:42.834649 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:42.832323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:04:49.577473 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:49.577437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:04:52.832698 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:04:52.832657    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:02.832144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:02.832055    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:03.814412 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:03.814372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:05:07.262439 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.262409    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:05:07.379785 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.379760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn94q\" (UniqueName: \"kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q\") pod \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") "
Sep 09 22:05:07.379966 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.379790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs\") pod \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\" (UID: \"71d135ee-8fdd-44d7-844e-a8a09fe0ea36\") "
Sep 09 22:05:07.382051 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.382021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q" (OuterVolumeSpecName: "kube-api-access-qn94q") pod "71d135ee-8fdd-44d7-844e-a8a09fe0ea36" (UID: "71d135ee-8fdd-44d7-844e-a8a09fe0ea36"). InnerVolumeSpecName "kube-api-access-qn94q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:07.382165 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.382027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71d135ee-8fdd-44d7-844e-a8a09fe0ea36" (UID: "71d135ee-8fdd-44d7-844e-a8a09fe0ea36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:07.480651 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.480620    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn94q\" (UniqueName: \"kubernetes.io/projected/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-kube-api-access-qn94q\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:05:07.480651 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.480648    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d135ee-8fdd-44d7-844e-a8a09fe0ea36-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:05:07.785476 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.785441    2563 generic.go:356] "Generic (PLEG): container finished" podID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerID="63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d" exitCode=0
Sep 09 22:05:07.785664 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.785512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"
Sep 09 22:05:07.785664 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.785531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" event={"ID":"71d135ee-8fdd-44d7-844e-a8a09fe0ea36","Type":"ContainerDied","Data":"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"}
Sep 09 22:05:07.785664 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.785596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5" event={"ID":"71d135ee-8fdd-44d7-844e-a8a09fe0ea36","Type":"ContainerDied","Data":"f5845387ff1234577f5f792ddc9ef585341ae529d949978a7a895570d4e6878a"}
Sep 09 22:05:07.785664 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.785626    2563 scope.go:117] "RemoveContainer" containerID="63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"
Sep 09 22:05:07.794478 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.794462    2563 scope.go:117] "RemoveContainer" containerID="63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"
Sep 09 22:05:07.794770 ip-10-0-138-22 kubenswrapper[2563]: E0909 22:05:07.794746    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d\": container with ID starting with 63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d not found: ID does not exist" containerID="63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"
Sep 09 22:05:07.794825 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.794773    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d"} err="failed to get container status \"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d\": rpc error: code = NotFound desc = could not find container \"63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d\": container with ID starting with 63d6a9d585e8a0b9b903f2f94b8cce748876bc0c7b4144e789b0a5e5b3c15b3d not found: ID does not exist"
Sep 09 22:05:07.802557 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.802535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:05:07.808570 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:07.808545    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ndsk5"]
Sep 09 22:05:09.577699 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:09.577664    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:09.718027 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:09.717975    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" path="/var/lib/kubelet/pods/71d135ee-8fdd-44d7-844e-a8a09fe0ea36/volumes"
Sep 09 22:05:18.833469 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:18.833424    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main" containerID="cri-o://f3bba18e0bea73031d1e0519f4b9fb98a79f1925da267106f29457cb9021cb22" gracePeriod=45
Sep 09 22:05:19.577639 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:19.577599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:29.577523 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:29.577466    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:29.578011 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:29.577582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:05:39.577562 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:39.577522    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 22:05:48.967245 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:48.967208    2563 generic.go:356] "Generic (PLEG): container finished" podID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerID="f3bba18e0bea73031d1e0519f4b9fb98a79f1925da267106f29457cb9021cb22" exitCode=0
Sep 09 22:05:48.967602 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:48.967281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" event={"ID":"902a62b1-705e-4ad1-93af-ecd33ff117c7","Type":"ContainerDied","Data":"f3bba18e0bea73031d1e0519f4b9fb98a79f1925da267106f29457cb9021cb22"}
Sep 09 22:05:49.084560 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.084534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:05:49.210664 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.210630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs\") pod \"902a62b1-705e-4ad1-93af-ecd33ff117c7\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") "
Sep 09 22:05:49.210847 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.210710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx2fk\" (UniqueName: \"kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk\") pod \"902a62b1-705e-4ad1-93af-ecd33ff117c7\" (UID: \"902a62b1-705e-4ad1-93af-ecd33ff117c7\") "
Sep 09 22:05:49.212854 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.212820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk" (OuterVolumeSpecName: "kube-api-access-vx2fk") pod "902a62b1-705e-4ad1-93af-ecd33ff117c7" (UID: "902a62b1-705e-4ad1-93af-ecd33ff117c7"). InnerVolumeSpecName "kube-api-access-vx2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:49.212975 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.212879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "902a62b1-705e-4ad1-93af-ecd33ff117c7" (UID: "902a62b1-705e-4ad1-93af-ecd33ff117c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:49.311913 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.311885    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902a62b1-705e-4ad1-93af-ecd33ff117c7-tls-certs\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:05:49.311913 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.311909    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2fk\" (UniqueName: \"kubernetes.io/projected/902a62b1-705e-4ad1-93af-ecd33ff117c7-kube-api-access-vx2fk\") on node \"ip-10-0-138-22.ec2.internal\" DevicePath \"\""
Sep 09 22:05:49.972670 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.972585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"
Sep 09 22:05:49.973130 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.972583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2" event={"ID":"902a62b1-705e-4ad1-93af-ecd33ff117c7","Type":"ContainerDied","Data":"92e91b7b7ddf6f2f8ab6de6d9a619eb37eaf4611f3f57b2621cfc0ab8c44887b"}
Sep 09 22:05:49.973130 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.972745    2563 scope.go:117] "RemoveContainer" containerID="f3bba18e0bea73031d1e0519f4b9fb98a79f1925da267106f29457cb9021cb22"
Sep 09 22:05:49.992070 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.992041    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:05:49.995173 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:49.995148    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8vmf2"]
Sep 09 22:05:51.715148 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:05:51.715110    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" path="/var/lib/kubelet/pods/902a62b1-705e-4ad1-93af-ecd33ff117c7/volumes"
Sep 09 22:06:20.244979 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:20.244944    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:06:21.063406 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:21.063376    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:06:21.874275 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:21.874243    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:06:22.804936 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:22.804906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:06:24.255249 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:24.255213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:24.269011 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:24.268983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:24.425446 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:24.425419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:25.878652 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:25.878617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:25.893872 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:25.893851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:26.049614 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:26.049544    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:27.480302 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:27.480266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:27.497254 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:27.497228    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:27.646316 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:27.646289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:29.043893 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:29.043860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:29.058250 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:29.058226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:29.213537 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:29.213504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:30.619410 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:30.619366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:30.633877 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:30.633857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:30.780804 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:30.780773    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:32.181319 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:32.181289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:32.197719 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:32.197691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:32.348082 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:32.348051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:33.745152 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:33.745117    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:33.759152 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:33.759128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:33.921263 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:33.921240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:35.314579 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:35.314548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:35.330032 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:35.330009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:35.480036 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:35.480007    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:36.918194 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:36.918162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:36.936438 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:36.936410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:37.091787 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:37.091754    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:38.502538 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:38.502505    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:38.516322 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:38.516297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:38.664653 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:38.664621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:40.057683 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:40.057653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:40.071324 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:40.071294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:40.215796 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:40.215765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:41.600911 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:41.600876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:41.614187 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:41.614157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:41.762703 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:41.762671    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:43.156424 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:43.156394    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:43.171164 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:43.171139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:43.325919 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:43.325889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:44.718890 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:44.718857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:44.733720 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:44.733692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:44.885039 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:44.885010    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:46.256257 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:46.256220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:46.270216 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:46.270191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:46.434243 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:46.434211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:47.842267 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:47.842233    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:47.855197 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:47.855169    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:48.008344 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:48.008313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:49.397740 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:49.397711    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:49.415427 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:49.415397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:49.568897 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:49.568867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:50.943279 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:50.943251    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:50.958360 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:50.958335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:51.101227 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:51.101185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:52.494774 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:52.494743    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:52.509910 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:52.509888    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:52.663968 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:52.663940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:54.033825 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:54.033796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:54.046574 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:54.046553    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:54.203518 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:54.203473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:55.574371 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:55.574341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:55.588855 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:55.588806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:55.766970 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:55.766943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:57.168607 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:57.168572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:57.181841 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:57.181816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:57.346991 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:57.346940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:06:58.735220 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:58.735183    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:06:58.755152 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:58.755126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:06:58.916506 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:06:58.916445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:07:00.260803 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:00.260772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:07:00.276116 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:00.276085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:07:00.434225 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:00.434193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:07:01.835144 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:01.835114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:07:01.850423 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:01.850398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:07:02.005568 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:02.005542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:07:08.487106 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:08.487073    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bbl9s_094fead8-5464-41e3-ab93-c0df81d66b51/global-pull-secret-syncer/0.log"
Sep 09 22:07:08.619467 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:08.619435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f6scw_83667412-d649-477f-b65c-14f3cc0fc2dc/konnectivity-agent/0.log"
Sep 09 22:07:08.714581 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:08.714552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-22.ec2.internal_3e31ec0601184593e40e8b698041193b/haproxy/0.log"
Sep 09 22:07:14.017550 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:14.017516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-6gzgs_d52ca83e-146d-49b0-a4e8-3df24f7bdc69/manager/0.log"
Sep 09 22:07:14.052813 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:14.052789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6vz6s_9cc11e15-865e-4993-9986-224da27d9728/manager/0.log"
Sep 09 22:07:14.283639 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:14.283559    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wd2k5_dbeda535-c26d-4595-95b7-7935e4295432/manager/0.log"
Sep 09 22:07:15.622226 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:15.619844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-lpfrg_7615b056-272c-4d62-9ea5-37861f719f42/cluster-monitoring-operator/0.log"
Sep 09 22:07:15.868816 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:15.868787    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48g7d_f13d6fae-3342-4ee3-ad5a-bd651718ea87/node-exporter/0.log"
Sep 09 22:07:15.905088 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:15.905013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48g7d_f13d6fae-3342-4ee3-ad5a-bd651718ea87/kube-rbac-proxy/0.log"
Sep 09 22:07:15.927902 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:15.927879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48g7d_f13d6fae-3342-4ee3-ad5a-bd651718ea87/init-textfile/0.log"
Sep 09 22:07:18.361556 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:18.361521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/1.log"
Sep 09 22:07:18.366013 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:18.365983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-99dnc_72802e85-2836-4fc1-950d-34f43feea95c/console-operator/2.log"
Sep 09 22:07:19.149207 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149170    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"]
Sep 09 22:07:19.149503 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main"
Sep 09 22:07:19.149503 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149501    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main"
Sep 09 22:07:19.149633 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149509    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main"
Sep 09 22:07:19.149633 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149514    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main"
Sep 09 22:07:19.149633 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149586    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d135ee-8fdd-44d7-844e-a8a09fe0ea36" containerName="main"
Sep 09 22:07:19.149633 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.149596    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="902a62b1-705e-4ad1-93af-ecd33ff117c7" containerName="main"
Sep 09 22:07:19.152671 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.152649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.155225 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.155205    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"kube-root-ca.crt\""
Sep 09 22:07:19.156363 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.156344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tw9dj\"/\"default-dockercfg-st6zb\""
Sep 09 22:07:19.156454 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.156343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"openshift-service-ca.crt\""
Sep 09 22:07:19.161741 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.161719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"]
Sep 09 22:07:19.255572 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.255537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-podres\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.255729 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.255575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjfp\" (UniqueName: \"kubernetes.io/projected/836acca3-408b-40bf-bbaf-dd5456387013-kube-api-access-nxjfp\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.255729 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.255675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-proc\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.255827 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.255733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-sys\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.255827 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.255757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-lib-modules\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356660 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjfp\" (UniqueName: \"kubernetes.io/projected/836acca3-408b-40bf-bbaf-dd5456387013-kube-api-access-nxjfp\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-proc\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-sys\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-lib-modules\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-proc\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-podres\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.356845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-sys\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.357111 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-lib-modules\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.357111 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.356896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/836acca3-408b-40bf-bbaf-dd5456387013-podres\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.365462 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.365438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjfp\" (UniqueName: \"kubernetes.io/projected/836acca3-408b-40bf-bbaf-dd5456387013-kube-api-access-nxjfp\") pod \"perf-node-gather-daemonset-6vfxg\" (UID: \"836acca3-408b-40bf-bbaf-dd5456387013\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.462814 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.462731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:19.471695 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.471668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-4qs9l_bbe35512-fe1b-4791-888a-d29338578816/volume-data-source-validator/0.log"
Sep 09 22:07:19.735194 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:19.735171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"]
Sep 09 22:07:19.737306 ip-10-0-138-22 kubenswrapper[2563]: W0909 22:07:19.737280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod836acca3_408b_40bf_bbaf_dd5456387013.slice/crio-65ef90856b153bd6bdde234226bc8608dd1807bc477e1a4b6fd21a968c9337b2 WatchSource:0}: Error finding container 65ef90856b153bd6bdde234226bc8608dd1807bc477e1a4b6fd21a968c9337b2: Status 404 returned error can't find the container with id 65ef90856b153bd6bdde234226bc8608dd1807bc477e1a4b6fd21a968c9337b2
Sep 09 22:07:20.358220 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.358179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg" event={"ID":"836acca3-408b-40bf-bbaf-dd5456387013","Type":"ContainerStarted","Data":"6fb23b5766610deb098d8a8606ed00a0c92e40cc66735befb43a32d7961bcd0f"}
Sep 09 22:07:20.358413 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.358241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg" event={"ID":"836acca3-408b-40bf-bbaf-dd5456387013","Type":"ContainerStarted","Data":"65ef90856b153bd6bdde234226bc8608dd1807bc477e1a4b6fd21a968c9337b2"}
Sep 09 22:07:20.358413 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.358385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:20.376744 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.376699    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg" podStartSLOduration=1.3766855740000001 podStartE2EDuration="1.376685574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:20.375578618 +0000 UTC m=+4457.127123526" watchObservedRunningTime="2026-09-09 22:07:20.376685574 +0000 UTC m=+4457.128230512"
Sep 09 22:07:20.640965 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.640928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjgzb_1fe439eb-c21c-4593-b642-c613c44889bd/dns/0.log"
Sep 09 22:07:20.662678 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.662652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjgzb_1fe439eb-c21c-4593-b642-c613c44889bd/kube-rbac-proxy/0.log"
Sep 09 22:07:20.688219 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:20.688197    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86nnz_8c8a7501-4a85-41d6-8cb2-b53133ab9993/dns-node-resolver/0.log"
Sep 09 22:07:21.275828 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:21.275801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dd7b9cfb-8qvq8_e602467d-8a1d-49b1-bba7-26c731030f1f/registry/0.log"
Sep 09 22:07:21.324459 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:21.324424    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g477p_30921f88-06ce-4cfd-aa6a-07542f3e82a6/node-ca/0.log"
Sep 09 22:07:22.383223 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:22.383189    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:07:23.069679 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.069647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xmv6d_9e374074-9180-4a2a-91b4-36f06f306cfe/serve-healthcheck-canary/0.log"
Sep 09 22:07:23.540178 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.540138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fckws_f146f95f-c995-445f-9d54-dc3931306c22/insights-operator/0.log"
Sep 09 22:07:23.541454 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.541433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fckws_f146f95f-c995-445f-9d54-dc3931306c22/insights-operator/1.log"
Sep 09 22:07:23.565145 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.565124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2g946_d03dc5cf-51f0-4aad-8ed6-00b7916e1f65/kube-rbac-proxy/0.log"
Sep 09 22:07:23.587285 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.587267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2g946_d03dc5cf-51f0-4aad-8ed6-00b7916e1f65/exporter/0.log"
Sep 09 22:07:23.613162 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:23.613144    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2g946_d03dc5cf-51f0-4aad-8ed6-00b7916e1f65/extractor/0.log"
Sep 09 22:07:26.372513 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:26.372470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-6vfxg"
Sep 09 22:07:27.581342 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:27.581315    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-644949856f-6jxms_7a2f40c9-e4f0-4a6e-abdd-50e1acb26af9/manager/0.log"
Sep 09 22:07:27.955275 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:27.955233    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-577669d9c7-2q7rb_07088d44-ad10-4762-87a9-4f93a987986e/manager/0.log"
Sep 09 22:07:27.978998 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:27.978971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-snjp9_58af9654-5576-4fd5-b026-280966228ce6/server/0.log"
Sep 09 22:07:28.635087 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:28.635058    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:07:28.836631 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:28.836601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-tb6vd_839862e4-4429-4dfe-bac8-1f3c78751267/manager/0.log"
Sep 09 22:07:28.935360 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:28.935326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qzpg4_c82d826e-6595-4db5-a8d4-538ed9c5a239/seaweedfs/0.log"
Sep 09 22:07:29.951737 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:29.951707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2gvhz_0600ee42-4cfa-4b14-8833-1ca6e1e56bc2/discovery/0.log"
Sep 09 22:07:35.444711 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.444676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4rgv9_76b32bd2-0309-478f-bc8f-6ffd579ea13f/kube-multus/0.log"
Sep 09 22:07:35.696203 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.696129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:07:35.721283 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.721257    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/egress-router-binary-copy/0.log"
Sep 09 22:07:35.743316 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.743293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/cni-plugins/0.log"
Sep 09 22:07:35.765356 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.765335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/bond-cni-plugin/0.log"
Sep 09 22:07:35.791063 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.791044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/routeoverride-cni/0.log"
Sep 09 22:07:35.812836 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.812819    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/whereabouts-cni-bincopy/0.log"
Sep 09 22:07:35.836129 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:35.836112    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mg6s9_faa80f37-df69-406b-88f5-f74e35d7ac63/whereabouts-cni/0.log"
Sep 09 22:07:36.111733 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:36.111659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rd976_7bf54879-0091-4062-a4b2-22ba7cbfacc3/network-metrics-daemon/0.log"
Sep 09 22:07:36.131830 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:36.131796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rd976_7bf54879-0091-4062-a4b2-22ba7cbfacc3/kube-rbac-proxy/0.log"
Sep 09 22:07:36.985353 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:36.985324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/ovn-controller/0.log"
Sep 09 22:07:37.003518 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.003474    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/ovn-acl-logging/0.log"
Sep 09 22:07:37.026356 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.026328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/kube-rbac-proxy-node/0.log"
Sep 09 22:07:37.046761 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.046736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 22:07:37.068312 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.068287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/northd/0.log"
Sep 09 22:07:37.089845 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.089812    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/nbdb/0.log"
Sep 09 22:07:37.112608 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.112585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/sbdb/0.log"
Sep 09 22:07:37.174502 ip-10-0-138-22 kubenswrapper[2563]: I0909 22:07:37.174468    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88qn5_90e54b4a-e9db-4412-ab99-bc597a89f378/ovnkube-controller/0.log"